Skip to content

Recommended Way to Restart Service Pods

To ensure safe and consistent behavior when restarting service pods, it is recommended to use a rolling restart via the kubectl rollout restart command. This method aligns with Kubernetes best practices and helps prevent potential data or state corruption.

For more detailed information, check Kubernetes Official Docs – kubectl rollout restart.

Example command

Check below command for example :

Bash
1
kubectl rollout restart deployment <service-name> -n <namespace>

This command will:

  • Restart pods one at a time (when using RollingUpdate strategy)
  • Ensure the new pod starts only after the old one has fully terminated
  • Maintain application availability during the restart process

Key Considerations

  • The benefits of a rolling restart (minimal downtime, no data corruption) are only applied when your deployment uses strategy: RollingUpdate.
  • If your deployment uses strategy: Recreate, kubectl rollout restart will delete all existing pods before creating new ones. In this scenario, it offers no availability advantage over manual pod deletion.

Avoid Manually Deleting Pods

It is strongly advised to avoid manually deleting pods using commands such as kubectl delete pod <pod-name>. Manual deletion can lead to:

  • Race conditions or unstable behavior: The new pod might start before the old one is completely terminated.
  • Potential data corruption or state inconsistencies: Improper termination can corrupt data or lead to an inconsistent state.
  • Service disruption and downtime: Uncontrolled deletion can cause unexpected service outages.

Connector-Specific Considerations

For connector services that manage their state using RocksDB, manual pod deletion is particularly risky. It can result in:

  • RocksDB corruption: Due to improper shutdown sequences.
  • Loss of maintained state and configuration data: Critical information might be lost.
  • Inconsistent connector behavior and potential data loss: The connector may not function correctly, leading to data issues.

Note

Connector deployments typically use strategy: Recreate. This is because running multiple connector pods simultaneously can cause RocksDB corruption due to concurrent access to the same database files. The Recreate strategy ensures that only one pod is running at a time, preventing this issue. Using a rolling restart ensures clean pod termination and startup sequencing, which reduces the risk of data corruption and promotes stable behavior.

To understand which Privacera services use the RollingUpdate or Recreate strategy, see Privacera Services Deployment Strategies.

Comments