PolicySync Changelog Slow-Task Diagnostics¶

PolicySync includes opt-in, off-by-default diagnostics that help find the cause when a changelog occasionally takes far longer to process than its work warrants (for example, a single resource task that appears to hang). The diagnostics are part of the shared PolicySync changelog path, so they apply to every connector.

When enabled, the connector captures — on its own, without anyone attaching a debugger:

• a slow-task WARN (to a dedicated file) naming the resource and the per-phase millisecond breakdown of any changelog resource task that runs longer than a threshold, and
• an automatic in-process thread dump when a changelog's processing exceeds a threshold, so the stuck thread's stack is recorded while it is still stuck.

Both features are disabled by default (their thresholds are 0). On upgrade nothing changes and nothing is written; enabling them is an explicit operator decision, and the per-task overhead when disabled is a single comparison.

Configuration¶

Set these in Privacera Manager. The primary thresholds — the slow-task WARN, the thread dump, and the resource-task time-box — all default to 0 (disabled) for every connector. You can enable them globally (set the variable so all connectors pick it up) or per connector instance — set the variable in that connector instance's custom vars (config/custom-vars/connectors/<connector>/<instance>/), which overrides the global default for that instance only. For example, enable the thread dump on the Databricks Unity Catalog instance while every other connector (Lake Formation, Snowflake, and so on) stays off.

The six stall.dump.* properties only take effect when the thread dump is enabled. The three resource.task.* properties are the independent resource-process task time-box, which is active only when policysync.changelog.resource.task.timeout.ms is non-zero. It uses two separate budgets — a processing budget measured from run-start (so a task merely waiting its turn in a deep queue is never abandoned as if it were stuck) and a queue budget measured from submit (the safety valve that releases the barrier when the pool is jammed). Size the queue budget above the expected queue depth when a small pool fans out many resource tasks.

When to enable¶

Enable these when a changelog is reported to be intermittently slow and you need to catch it in the act — the slowness is usually random, so attaching a debugger after the fact rarely works. Because everything is off by default, you turn on only what you need and turn it back off once you have collected the evidence.

Operator runbook¶

1. Enable the diagnostics. In Privacera Manager, set the thresholds to the recommended values, then run the Privacera Manager update so the connector picks up the new policysync.properties:

   YAML
   1 2	CONNECTOR_CHANGELOG_SLOW_RESOURCE_TASK_WARN_MS: "30000" CONNECTOR_CHANGELOG_SLOW_CHANGELOG_DUMP_MS: "600000"

   Optionally, to also time-box an over-running resource task (abandon + log, release the barrier, and emit a RESOURCE_PROCESS_TASK_TIMEOUT audit after retries):

   YAML
   1 2 3	CONNECTOR_CHANGELOG_RESOURCE_TASK_TIMEOUT_MS: "300000" CONNECTOR_CHANGELOG_RESOURCE_TASK_QUEUE_TIMEOUT_MS: "3600000" CONNECTOR_CHANGELOG_RESOURCE_TASK_MAX_RETRIES: "1"

2. Reproduce, or wait for recurrence. Let the connector run until a slow changelog occurs again.

3. Collect the output from the connector pod. The diagnostics write to the connector's log directory:

   • changelog-slow-task.log — one WARN line per slow resource task, with its resource key, resource type, principal count, and the per-phase millisecond breakdown. This file is written by a dedicated, file-only logger and is not sent to the console or logstash appenders.
   • dump/<YYYY-MM-DD>/<changeLogId>/ — the captured thread dumps (dump-00.txt, and so on) plus a summary file listing the in-flight resource keys and the step each was executing when the dump was taken.

   Bash
   1 2 3 4	kubectl exec -it <POD_NAME> -n <NAMESPACE> -- /bin/bash # inside the pod, under the connector log directory: ls -R dump/ cat changelog-slow-task.log

   See Capturing a Thread Dump and Collecting Support Files for copying these files off the pod along with policysync.properties / connector.properties.

4. Interpret the output. The WARN's per-phase breakdown tells you which phase of the task was slow; the summary file names the step each in-flight task was on; and the thread dumps show the exact stack of the stuck thread. Together these localize where the time went.

5. Disable when done. Set the thresholds back to 0 and re-apply (include the time-box's CONNECTOR_CHANGELOG_RESOURCE_TASK_TIMEOUT_MS and CONNECTOR_CHANGELOG_RESOURCE_TASK_QUEUE_TIMEOUT_MS if you enabled them):

   YAML
   1 2 3 4	CONNECTOR_CHANGELOG_SLOW_RESOURCE_TASK_WARN_MS: "0" CONNECTOR_CHANGELOG_SLOW_CHANGELOG_DUMP_MS: "0" CONNECTOR_CHANGELOG_RESOURCE_TASK_TIMEOUT_MS: "0" CONNECTOR_CHANGELOG_RESOURCE_TASK_QUEUE_TIMEOUT_MS: "0"