Skip to content

Migrate from ZooKeeper mode to KRaft mode for Kafka

This guide outlines the steps to migrate Kafka from ZooKeeper mode to KRaft mode. KRaft (Kafka Raft) is Apache Kafka's consensus protocol that removes the dependency on ZooKeeper for metadata management.

Tip

KRaft mode is supported starting with Privacera Platform version 9.0.32.1. For earlier versions, you must use ZooKeeper mode.

Note

Estimated Downtime: This migration will require downtime for Kafka and associated Discovery services. The duration depends on the completion of any active Discovery scans and the time needed to perform the upgrade steps, typically ranging from 20 minutes to 1 hour. Plan accordingly, and ensure that any critical Discovery tasks are paused or completed prior to the migration window.

Migration Steps

Step 1: Stop Active Discovery Scans

a. Stop existing offline Discovery scans:

  • Log in to the Privacera Platform portal
  • Navigate to Discovery Menu -> Scan Status.
  • Stop, kill, or cancel any existing offline Discovery scans that are currently running, listing, or pending

Note

If you do not wish to cancel any existing offline scans, wait until they have completed before proceeding with the upgrade.

b. Stop the PKafka Pod to temporarily pause Real-Time Scans:

Bash
1
kubectl -n <name-space> scale deploy --replicas=0 pkafka

Tip

Replace with your Kubernetes namespace. This is usually set using the DEPLOYMENT_ENV_NAME environment variable.

Step 2: Wait for Active Scans to Complete

  • Run the following command periodically and monitor its output. An output value of zero signifies that there are no active discovery scan executor pods remaining.
Bash
1
kubectl -n <name-space> get pods | grep "exec" | wc -l
  • Wait until the output of the command is 0 before proceeding to the next step.

Step 3: Scale Down Privacera Services

Scale down the following services before proceeding with the upgrade:

  • Discovery Service:

    Bash
    1
    kubectl -n <name-space> scale deploy --replicas=0 discovery

  • Discovery Consumer:

    Bash
    1
    kubectl -n <name-space> scale deploy --replicas=0 discovery-consumer

  • Kafka Pod:

    Bash
    1
    kubectl -n <name-space> scale sts --replicas=0 kafka

Step 4: Delete Existing Kafka StatefulSet

Once all services are scaled down, delete the existing Kafka StatefulSet.

Bash
1
kubectl -n <name-space> delete sts kafka

Step 5: Clean Up Existing Storage

Delete the existing PVC (Persistent Volume Claim):

Note

Run kubectl -n <name-space> get pvc | grep datadir-kafka to list all Kafka PVCs. If your Kafka cluster has multiple replicas, you must delete each PVC.

Bash
1
kubectl -n <name-space> delete pvc datadir-kafka-0

Step 6: Enable KRaft Mode

  • Edit the Kafka configuration file:

    Bash
    1
2
    cd ~/privacera/privacera-manager
vi config/custom-vars/vars.kafka.yml

  • Add the KRaft mode property:

    YAML
    1
2
    # Enable KRaft mode for Kafka
KAFKA_ENABLE_KRAFT_MODE: "true"

Step 7: Restart Privacera Services

Restart services and post install:

Bash
1
2
3
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade

Run the following command to run the post install steps:

Bash
1
2
cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Step 8: Verify Kafka Migration

Check Kafka pods are running:

Bash
1
kubectl -n <name-space> get pods | grep kafka

Post-Migration Verification

Test Discovery functionality:

  • Run a Discovery scan to confirm that Discovery works as expected.
  • If Discovery functionality is not working, use the Privacera diagnostic tool for debugging and log analysis.

Comments