Migrate from ZooKeeper mode to KRaft mode for Kafka¶

This guide outlines the steps to migrate Kafka from ZooKeeper mode to KRaft mode. KRaft (Kafka Raft) is Apache Kafka's consensus protocol that removes the dependency on ZooKeeper for metadata management.

Supported Kafka Modes¶

The following table defines the supported metadata modes based on your Privacera Platform version:

Before proceeding, review the Breaking change and Known limitations sections below to confirm the steps that apply to your version.

Migration Steps¶

Step 1: Stop Active Discovery Scans¶

a. Stop existing offline Discovery scans:

• Log in to the Privacera Platform portal
• Navigate to Discovery Menu -> Scan Status.
• Stop, kill, or cancel any existing offline Discovery scans that are currently running, listing, or pending

b. Stop the PKafka Pod to temporarily pause Real-Time Scans:

1

kubectl -n <name-space> scale deploy --replicas=0 pkafka

Step 2: Wait for Active Scans to Complete¶

• Run the following command periodically and monitor its output. An output value of zero signifies that there are no active discovery scan executor pods remaining.

1

kubectl -n <name-space> get pods | grep "exec" | wc -l

• Wait until the output of the command is 0 before proceeding to the next step.

Optional: See Kafka backlog before you scale Kafka down¶

Use this step to verify whether any consumers still have unread messages before shutting down Kafka.

When to run: After Step 2 (active scans complete), before Step 3 (Kafka must still be running).

Run this command to check whether any consumer groups have unprocessed messages on the cluster:

Replace <name-space> with your Kubernetes namespace.

1
2
3
4
5
6
7
8

kubectl -n <name-space> exec -it kafka-0 -c k8skafka -- \
  sh -c 'kafka-consumer-groups.sh \
    --bootstrap-server kafka-ha:9092 \
    --describe \
    --all-groups 2>/dev/null' \
| awk '/^GROUP[[:blank:]]+TOPIC[[:blank:]]+PARTITION/ {next}
       NF >= 6 && $3 ~ /^[0-9]+$/ {print $2 "\t" $6}' \
| column -t -s $'\t'

What you see: Each line shows a topic name and its LAG value — the number of messages waiting to be read by consumer groups. A LAG of 0 means that consumer group is fully caught up on that partition.

If the command fails, try these fixes in order:

Step 3: Scale Down Privacera Services¶

Scale down the following services before proceeding with the upgrade:

• Discovery Service:

  Bash
  1	kubectl -n <name-space> scale deploy --replicas=0 discovery

• Discovery Consumer:

  Bash
  1	kubectl -n <name-space> scale deploy --replicas=0 discovery-consumer

• Kafka Pod:

  Bash
  1	kubectl -n <name-space> scale sts --replicas=0 kafka

Step 4: Delete Existing Kafka StatefulSet¶

Once all services are scaled down, delete the existing Kafka StatefulSet.

1

kubectl -n <name-space> delete sts kafka

Step 5: Clean Up Existing Storage¶

Delete the existing PVC (Persistent Volume Claim):

1

kubectl -n <name-space> delete pvc datadir-kafka-0

Step 6: Enable KRaft Mode (9.0.32.1 through 9.2.18.1 only)¶

For 9.0.32.1 through 9.2.18.1 only:

• Edit the Kafka configuration file:

  Bash
  1 2	cd ~/privacera/privacera-manager vi config/custom-vars/vars.kafka.yml

• Add the KRaft mode property:

  YAML
  1 2	# Enable KRaft mode for Kafka KAFKA_ENABLE_KRAFT_MODE: "true"

Step 7: Restart Privacera Services¶

Restart services and post install:

1
2
3

cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade 

Run the following command to run the post install steps:

1
2

cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Step 8: Verify Kafka Migration¶

Check Kafka pods are running:

1

kubectl -n <name-space> get pods | grep kafka

Post-Migration Verification¶

Test Discovery functionality:

• Run a Discovery scan to confirm that Discovery works as expected.
• If Discovery functionality is not working, use the Privacera diagnostic tool for debugging and log analysis.