Advanced On-Demand Task Configuration¶

This document describes advanced configuration options for on-demand task processing in PolicySync connectors. It covers dynamic polling and task cleanup on restart.

Dynamic Polling: Automatically adjusts polling frequency based on task availability
Task Cleanup on Restart: Prevents duplicate processing after connector restarts

Prerequisites¶

Before configuring these advanced features, ensure that on-demand sync is enabled for your PolicySync connector:

On Demand Sync for PolicySync Connectors

Dynamic Polling Configuration¶

Dynamic polling allows connectors to automatically adjust their polling frequency based on the availability of new tasks. This optimization significantly reduces server load during idle periods while maintaining responsiveness when tasks are actively processed.

Configuration Properties¶

Variable	Default	Description
`CONNECTOR_ON_DEMAND_TASK_DYNAMIC_POLLING_ENABLED`	`false`	Enables dynamic polling with exponential backoff for on-demand tasks
`CONNECTOR_ON_DEMAND_TASK_MIN_POLLING_INTERVAL_SEC`	`1`	Minimum polling interval in seconds when tasks are available (fast polling rate)
`CONNECTOR_ON_DEMAND_TASK_SYNC_INTERNAL_SEC`	`30`	Maximum polling interval in seconds (acts as ceiling for dynamic polling)

How Dynamic Polling Works¶

When dynamic polling is enabled (CONNECTOR_ON_DEMAND_TASK_DYNAMIC_POLLING_ENABLED: "true"):

Fast Polling: When tasks are detected, the system polls continuously at the minimum interval (CONNECTOR_ON_DEMAND_TASK_MIN_POLLING_INTERVAL_SEC).
Exponential Backoff: When no tasks are found, the polling interval gradually increases using exponential backoff
Maximum Ceiling: The polling interval never exceeds the value set in CONNECTOR_ON_DEMAND_TASK_SYNC_INTERNAL_SEC
Reset to Fast: As soon as a task becomes available, polling immediately resets to the minimum interval.

When dynamic polling is disabled (CONNECTOR_ON_DEMAND_TASK_DYNAMIC_POLLING_ENABLED: "false"):

The connector polls at a fixed interval specified by CONNECTOR_ON_DEMAND_TASK_SYNC_INTERNAL_SEC (default: 30 seconds)
The polling frequency remains constant, regardless of task availability
This is the traditional, backward-compatible behavior

Important Notes¶

Configuration Validation

Ensure CONNECTOR_ON_DEMAND_TASK_MIN_POLLING_INTERVAL_SEC is less than CONNECTOR_ON_DEMAND_TASK_SYNC_INTERNAL_SEC. If not, dynamic polling will be automatically disabled.

Performance Optimization

Enabling Dynamic polling is recommended for better resource utilization, as it significantly reduces unnecessary server load during idle periods.

Configuration for Different Deployments¶

Default Values

By default, dynamic polling is disabled (false).

To enable dynamic polling, add the following custom properties to your connector configuration:

Self Managed (Portal Configuration)Self Managed (YAML Configuration) and Data Plane

Navigate to Settings → Applications in the Self-Managed Portal.
From the list of Connected Applications, select your connector.
Click on the application name or the icon to edit. Then, go to the Access Management tab.

Under Add New Custom Properties add the following properties to enable dynamic polling:

Bash
policysync.on-demand.task.dynamic.polling.enabled=true
policysync.on-demand.task.min.polling.interval.sec=1
policysync.on-demand.task.sync.interval.sec=30

Click SAVE to apply the changes.

SSH to the instance where Privacera Manager is installed.
Run the following command to open the connector .yml file to be edited.

If you have multiple connectors, then replace the path with the appropriate connector instance name.
Bash
```
vi ~/privacera/privacera-manager/config/custom-vars/connectors/<connector-type>/<instance>/vars.connector.<connector-type>.yml
```

Add the following properties to enable dynamic polling:

YAML
CONNECTOR_ON_DEMAND_TASK_DYNAMIC_POLLING_ENABLED: "true"
CONNECTOR_ON_DEMAND_TASK_MIN_POLLING_INTERVAL_SEC: "1"
CONNECTOR_ON_DEMAND_TASK_SYNC_INTERNAL_SEC: "30"

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1 2
cd ~/privacera/privacera-manager ./pm_with_helm.sh upgrade
Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

Task Cleanup on Restart¶

This feature automatically cancels pending on-demand tasks when a connector restarts, preventing duplicate processing after a full synchronization.

Configuration Properties¶

Variable	Default	Description
`CONNECTOR_ON_DEMAND_TASK_CLEANUP_ON_RESTART_ENABLED`	`false`	Enables automatic cancellation of pending tasks when connector restarts

How Task Cleanup Works¶

When a connector restarts, it performs a full sync that loads all resources. To prevent duplicate processing, accumulated pending tasks (NEW, WAITING) are automatically marked as CANCELLED.

Benefits¶

Prevents duplicate processing after full sync
Maintains clean task queue
Preserves audit trails (tasks marked as CANCELLED, not deleted)

Configuration for Different Deployments¶

Default Value

By default, task cleanup on restart is disabled (false).

Important Consideration

Enable this feature to prevent duplicate processing after restarts. Since a full sync already loads all resources, marking old tasks as CANCELLED ensures accuracy without losing audit history.

To enable this feature, add the following custom properties to your connector configuration:

Self Managed (Portal Configuration)Self Managed (YAML Configuration) and Data Plane

Navigate to Settings → Applications in the Self-Managed Portal.
From the list of Connected Applications, select your connector.
Click on the application name or the icon to edit. Then, go to the Access Management tab.
Under Add New Custom Properties add the following property to enable task cleanup on restart:
Bash
1
policysync.on-demand.task.cleanup.on.restart.enabled=true
Click SAVE to apply the changes.

SSH to the instance where Privacera Manager is installed.
Run the following command to open the connector .yml file to be edited.

If you have multiple connectors, then replace the path with the appropriate connector instance name.
Bash
```
vi ~/privacera/privacera-manager/config/custom-vars/connectors/<connector-type>/<instance>/vars.connector.<connector-type>.yml
```
Add the following property to enable task cleanup on restart:
YAML
1
CONNECTOR_ON_DEMAND_TASK_CLEANUP_ON_RESTART_ENABLED: "true"
Once the property is configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1 2
cd ~/privacera/privacera-manager ./pm_with_helm.sh upgrade
Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

Prev topic: Advance Configuration