Configure Kubernetes Pod Scheduling¶

This guide describes how to configure Kubernetes pod scheduling features for Privacera services. This allows you to control where pods are scheduled, how they tolerate node taints, and how they are distributed across your Kubernetes cluster for high availability.

Overview¶

Privacera Manager supports the following Kubernetes scheduling features:

Node Selectors: Control which nodes pods can be scheduled on based on node labels
Tolerations: Allow pods to schedule on nodes with matching taints
Node Affinity: Define advanced node selection rules with flexible matching
Pod Topology Spread Constraints: Control pod distribution across zones, nodes, or other topology domains
Pod Disruption Budgets: Ensure minimum pod availability during voluntary disruptions (e.g., node maintenance)

These features can be configured globally for all services or per-service for fine-grained control.

Prerequisites¶

Before configuring pod scheduling, ensure:

Kubernetes cluster with labeled nodes and/or taints configured
For cloud-specific configurations (AWS EKS with Karpenter, Azure AKS, GCP GKE), ensure node pools and labels are properly configured
Understanding of your cluster's node labels, taints, and topology domains

Configuration Steps¶

To configure pod scheduling features, follow the steps below:

1. Copy the Scheduling Configuration File¶

Bash
cd ~/privacera/privacera-manager
cp config/sample-vars/vars.kubernetes.scheduling.yml config/custom-vars/

This file contains all scheduling-related configurations including node selectors, tolerations, node affinity, topology spread constraints, and pod disruption budgets.

2. Configure Global Scheduling Settings (Optional)¶

Edit config/custom-vars/vars.kubernetes.scheduling.yml and uncomment the global settings to apply scheduling rules to all Privacera services.

Note: All values shown below are sample values for reference. Do not use these values directly. You must identify and use values specific to your Kubernetes environment and configuration (node labels, taints, topology domains, etc.).

Global Node Selector¶

To schedule all pods on nodes with specific labels:

YAML
K8S_NODE_SELECTOR_ENABLE: "true"
K8S_DEPLOYMENT_NODE_SELECTORS:
  karpenter.sh/nodepool: "privacera-nodepool"
  kubernetes.io/arch: "amd64"

Global Tolerations¶

To allow all pods to schedule on nodes with specific taints:

YAML
K8S_TOLERATION_ENABLE: "true"
K8S_TOLERATIONS:
  - key: "karpenter.sh/nodepool"
    operator: "Equal"
    value: "privacera-nodepool"
    effect: "NoSchedule"

Supported operators: Equal, Exists
Supported effects: NoSchedule, PreferNoSchedule, NoExecute

Global Node Affinity¶

For advanced node selection with multiple matching rules:

YAML
K8S_NODE_AFFINITY_ENABLE: "true"
K8S_NODE_AFFINITY_SCHEDULING_TYPE: "requiredDuringSchedulingIgnoredDuringExecution"
K8S_NODE_AFFINITY_MATCH_EXPRESSIONS:
  - key: "karpenter.sh/nodepool"
    operator: "In"
    values:
      - "privacera-nodepool"
  - key: "kubernetes.io/arch"
    operator: "In"
    values:
      - "amd64"

Scheduling Types:

requiredDuringSchedulingIgnoredDuringExecution: Hard requirement (pods won't schedule if not met)
preferredDuringSchedulingIgnoredDuringExecution: Soft preference (best effort)

Match Operators: In, NotIn, Exists, DoesNotExist, Gt, Lt

Global Pod Topology Spread¶

To distribute pods evenly across zones and nodes:

YAML
K8S_POD_TOPOLOGY_ENABLE: "true"
K8S_TOPOLOGIES:
  - topologykey: "topology.kubernetes.io/zone"
    maxskew: 1
    whenunsatisfiable: "ScheduleAnyway"
  - topologykey: "kubernetes.io/hostname"
    maxskew: 1
    whenunsatisfiable: "ScheduleAnyway"

Parameters:

topologykey: Node label representing the topology domain (zone, hostname, rack, etc.)
maxskew: Maximum allowed difference in pod count between topology domains
whenunsatisfiable: DoNotSchedule (hard) or ScheduleAnyway (soft)

Global Pod Disruption Budget¶

To protect services during voluntary disruptions:

YAML
K8S_POD_DISRUPTION_BUDGET_ENABLED: "true"

Note: This enables Pod Disruption Budgets with default specifications. To customize the PDB specification (minAvailable/maxUnavailable), use service-specific configuration (see below).

3. Configure Service-Specific Scheduling (Optional)¶

To override global settings or customize specific features for individual services, locate and uncomment the respective service configuration section in the file.

Example: Configure scheduling for Ranger Admin

Locate the Ranger Admin section in vars.kubernetes.scheduling.yml:

YAML
## Ranger Admin

#RANGER_NODE_SELECTOR_ENABLE: "true"
#RANGER_DEPLOYMENT_NODE_SELECTORS:
#  custom.label/key: "custom-value"

#RANGER_TOLERATION_ENABLE: "true"
#RANGER_TOLERATIONS:
#  - key: "custom.taint/key"
#    operator: "Equal"
#    value: "custom-value"
#    effect: "NoSchedule"

#RANGER_NODE_AFFINITY_ENABLE: "true"
#RANGER_NODE_AFFINITY_SCHEDULING_TYPE: "requiredDuringSchedulingIgnoredDuringExecution"
#RANGER_NODE_AFFINITY_MATCH_EXPRESSIONS:
#  - key: "custom.label/key"
#    operator: "In"
#    values:
#      - "custom-value"

#RANGER_K8S_POD_TOPOLOGY_ENABLE: "true"
#RANGER_TOPOLOGIES:
#  - topologykey: "topology.kubernetes.io/zone"
#    maxskew: 1
#    whenunsatisfiable: "ScheduleAnyway"
#  - topologykey: "kubernetes.io/hostname"
#    maxskew: 1
#    whenunsatisfiable: "ScheduleAnyway"

#RANGER_K8S_POD_DISRUPTION_BUDGET_ENABLED: "true"

## Override the default Pod Disruption Budget specification (uncomment only the required value)
#RANGER_K8S_POD_DISRUPTION_BUDGET_SPEC:
#  minAvailable: 2                  # Keep at least 2 pods available
#  maxUnavailable: 1                # Allow maximum 1 pod to be unavailable
#  minAvailable: "50%"              # Keep at least 50% of pods available
#  maxUnavailable: "25%"            # Allow maximum 25% of pods to be unavailable

Uncomment the lines you need and update the values based on your Kubernetes environment setup and configuration to make it work.

Pod Disruption Budget Customization:

The Pod Disruption Budget (PDB) specification is service-specific only and allows you to customize availability requirements during voluntary disruptions:

minAvailable: Minimum number (or percentage) of pods that must remain available
Example: minAvailable: 2 - Keep at least 2 pods running
Example: minAvailable: "50%" - Keep at least 50% of pods running
maxUnavailable: Maximum number (or percentage) of pods that can be unavailable
Example: maxUnavailable: 1 - Allow at most 1 pod to be unavailable
Example: maxUnavailable: "25%" - Allow at most 25% of pods to be unavailable

Note: Use either minAvailable or maxUnavailable, not both. Choose the option that best fits your availability requirements.

The configuration file contains similar sections for all Privacera services (Portal, Discovery, Kafka, Solr, etc.). Follow the same pattern: uncomment the relevant service section and modify values for your environment.

4. Deploy or Update Privacera Manager¶

After copying the configuration file and making necessary changes:

For new/fresh installation:

Bash
   cd ~/privacera/privacera-manager/
   ./privacera-manager setup  
   ./pm_with_helm.sh install  
   ./privacera-manager post-install

For upgrade scenario (existing installation):

Bash
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade 

Run the following command to run the post install steps:

Bash
cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

This will apply the scheduling configurations to the Privacera services.

5. Verify the Configuration¶

Once deployment completes, verify that the scheduling configurations are applied:

Check pod placement:

Bash
kubectl get pods -n <YOUR_NAMESPACE> -o wide

This shows which nodes the pods are running on.

Check pod scheduling details:

Bash
kubectl describe pod <POD_NAME> -n <YOUR_NAMESPACE>

Look for sections like:

Node-Selectors
Tolerations
Node Affinity
Topology Spread Constraints

Check pod disruption budgets:

Bash
kubectl get pdb -n <YOUR_NAMESPACE>

Troubleshooting¶

Pods Stuck in Pending State¶

If pods are not scheduling:

Check node labels:
Bash
1
kubectl get nodes --show-labels
Verify that your node selector labels exist on nodes.

Check node taints:

Bash
kubectl describe node <NODE_NAME> | grep -A 5 Taints

Ensure tolerations match the node taints.

Check pod events:
Bash
1
kubectl describe pod <POD_NAME> -n <YOUR_NAMESPACE>
Look for scheduling errors in the Events section.

Uneven Pod Distribution¶

If pods are not distributed as expected:

Check topology labels:
Bash
1
kubectl get nodes -L topology.kubernetes.io/zone
Verify topology labels exist on nodes.
Review topology configuration:
Lower maxskew for stricter distribution
Change whenunsatisfiable to DoNotSchedule for hard enforcement
Ensure sufficient nodes in each topology domain

Disabling Configurations¶

To disable any scheduling feature:

Identify the _ENABLE flag for the required service which you want to disable
Set it to "false" in the configuration file
Save the changes
Apply the changes:

Bash
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade 

Run the following command to run the post install steps:

Bash
cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Best Practices¶

Start with global configuration for consistent scheduling across all services
Use service-specific overrides only when needed (e.g., database pods on specific node types, custom PDB specifications)
Always enable topology spread constraints in production for high availability
Enable pod disruption budgets to protect against service interruptions during maintenance
Customize PDB specifications for critical services based on their replica counts and availability requirements
For services with 2 replicas: Use minAvailable: 1 or maxUnavailable: 1
For services with 3+ replicas: Use minAvailable: 2 or percentage-based values like maxUnavailable: "33%"
Test in non-production environments before applying to production
Document your node labels and taints for team reference
Use ScheduleAnyway for topology spread in initial setup, then switch to DoNotSchedule after verifying distribution

Additional Resources¶

Previous: Advanced Configuration