Schedule Data Zone Re-evaluation¶

Overview¶

When a file in a data zone is tagged, Discovery applies the data zone's workflow policy to it, for example moving or copying the file to a destination. If that step does not finish, the file stays tagged but is not moved. Re-evaluating the data zone applies the policy again and picks the file up. See Re-evaluating a Data Zone.

Instead of re-evaluating by hand, you can let Discovery re-evaluate a data source's data zones on a schedule. You turn this on per data source with one property, Periodic Datazone Re-evaluation Interval (Minutes, 0 = Off).

This is available from release 9.2.45.1.

Which data sources support it¶

The property is shown on cloud storage data sources:

• Amazon S3
• Google Cloud Storage (GCS)
• Azure Data Lake Storage (ADLS)

Database data sources do not have the property.

What a scheduled run does¶

On each run, Discovery looks at the data zones that include resources from this data source, and re-evaluates each data zone that has a workflow policy. This is the same work as selecting Re-evaluate on the data zone's resources:

• Discovery lists the files under each resource of the data zone.
• It applies the data zone's policies again to the files that Discovery has already scanned. A file that a workflow policy applies to, but that was not moved or copied yet, is moved or copied now.
• Alerts are matched by policy and resource, so a re-evaluation updates an existing alert and does not create a duplicate.

A scheduled run does not:

• Re-evaluate data zones without a workflow policy. Destination and quarantine data zones are left alone, so files that were already moved there are not processed again.
• Scan new files. Files that Discovery has not scanned yet are skipped until they are scanned.

Values¶

The first run happens one interval after Discovery sees the new value. After each run, the interval starts again, so a slow run does not cause runs to overlap. A change to the value takes effect within a few minutes, when Discovery next refreshes its configuration. You do not need to restart Discovery.

Turn on scheduled re-evaluation¶

1. Sign in to the Privacera Portal as an administrator.
2. Go to Settings > Data Source Registration, select your system, and open the cloud storage application, for example Google Cloud Storage.
3. Under Application properties, find Periodic Datazone Re-evaluation Interval (Minutes, 0 = Off) (datazone.reevaluate.schedule.interval.minutes).
4. Enter the number of minutes between runs, for example 60.
5. Select Save.

Repeat these steps for each data source that needs it. Each data source has its own value.

Choose an interval¶

Each run lists every file under the data zone's resources. For a data zone with many files, a short interval adds steady load on Discovery and on the cloud storage API. Choose the longest interval that still moves stranded files soon enough for your needs. 60 minutes is a reasonable starting point. Use shorter intervals, such as 5, only for testing.

Check that it is running¶

Each run writes one line to the Discovery log (the Discovery scanner pod, not the consumer pods):

1

Periodic datazone re-evaluation for appCode=<application-code> re-evaluated <resource-count> workflow datazone resource(s), <failed-count> failed, in <duration> ms

<resource-count> is the number of data zone resources re-evaluated, and <failed-count> is the number that failed. If <resource-count> is 0, the data source has no data zone with a workflow policy.

Discovery also reports these metrics, each with an appCode tag:

Turn off scheduled re-evaluation¶

Set the property back to 0 and select Save. Runs stop when Discovery next refreshes its configuration. Manual Re-evaluate in the data zone keeps working whatever the value is.