Skip to content

Solr Collection Retention (TTL) for Privacera Discovery

Overview

Privacera Discovery writes scan results (classified resources, lineage events, alerts, audits, telemetry, and operational logs) into a set of Solr collections that back the Data Discovery pages in the Privacera Portal. Document retention (TTL) controls how long each of those collections keeps its documents before they are automatically deleted.

Each Discovery collection has a product-default retention. You can shorten or lengthen retention on any collection, and you can turn retention on for collections that ship with it disabled, by setting a single Ansible variable in Privacera Manager.

How retention is applied

Retention is expressed in days per collection through the SOLR_COLLECTION_PRIVACERA_<name>_TTL_DAYS Ansible variable family. Privacera Manager applies the value at install time. Once set, expired documents are deleted automatically on a 24-hour sweep cycle.

Seven Discovery collections ship with retention enabled and a default value applied automatically. Five ship with retention disabled, and each can be enabled individually by setting its Ansible variable.

Collections with retention enabled by default

Collection Default value Ansible variable
privacera_logs 7 days SOLR_COLLECTION_PRIVACERA_LOGS_TTL_DAYS
privacera_offline_scan_summary 10 days SOLR_COLLECTION_PRIVACERA_OFFLINE_SCAN_SUMMARY_TTL_DAYS
privacera_offline_scan_cleanup 10 days SOLR_COLLECTION_PRIVACERA_OFFLINE_SCAN_CLEANUP_TTL_DAYS
privacera_metrics 14 days SOLR_COLLECTION_PRIVACERA_METRICS_TTL_DAYS
privacera_audits 31 days SOLR_COLLECTION_PRIVACERA_AUDITS_TTL_DAYS
privacera_spark_events 31 days SOLR_COLLECTION_PRIVACERA_SPARK_EVENTS_TTL_DAYS
privacera_discovery_telemetry 90 days SOLR_COLLECTION_PRIVACERA_DISCOVERY_TELEMETRY_TTL_DAYS

Collections with retention disabled by default

These collections keep documents indefinitely unless retention is explicitly enabled by setting the corresponding Ansible variable.

Collection Ansible variable to enable
privacera_alerts SOLR_COLLECTION_PRIVACERA_ALERTS_TTL_DAYS
privacera_classification SOLR_COLLECTION_PRIVACERA_CLASSIFICATION_TTL_DAYS
privacera_lineage SOLR_COLLECTION_PRIVACERA_LINEAGE_TTL_DAYS
privacera_resource SOLR_COLLECTION_PRIVACERA_RESOURCE_TTL_DAYS
privacera_resource_meta_info SOLR_COLLECTION_PRIVACERA_RESOURCE_META_INFO_TTL_DAYS

Changing retention

Retention is configured through Privacera Manager custom variables. Add or update the relevant entry in config/custom-vars/vars.solr.yml:

YAML
1
2
3
4
5
6
7
8
# Shorten a discovery collection's retention to 30 days
SOLR_COLLECTION_PRIVACERA_LOGS_TTL_DAYS: "30"

# Enable retention on an opt-in collection (60 days)
SOLR_COLLECTION_PRIVACERA_LINEAGE_TTL_DAYS: "60"

# Effectively disable retention on an on-by-default collection
SOLR_COLLECTION_PRIVACERA_AUDITS_TTL_DAYS: "36500"

Restart Privacera Services

After updating the retention values, restart Discovery:

Bash
1
2
3
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade

The following profiles are suggested starting points. Adjust individual values to match your legal, regulatory, and internal data-retention policies.

Collection Short Balanced Compliance-heavy
privacera_classification 90 180 365
privacera_resource 90 180 365
privacera_resource_meta_info 90 180 365
privacera_lineage 180 365 730
privacera_alerts 30 90 180
privacera_offline_scan_summary 90 180 365
privacera_offline_scan_cleanup 3 7 10
privacera_logs 7 14 30
privacera_metrics 14 30 90
privacera_spark_events 14 31 90
privacera_audits 90 180 365
privacera_discovery_telemetry 30 90 180

Collection catalog

The table below describes each Discovery Solr collection and the Portal screens it populates.

Collection What is stored Portal screens populated
privacera_classification Classified resources: tags, snippets, classification status, application, data zone, scores, and resource metadata. Classifications; Reviews; Data Explorer; Resource Detail; Dashboard KPI tiles; Data Zone Dashboard; most classification reports.
privacera_resource Resource-level scan counts (files, folders, tables, columns scanned), including untagged resources used for totals. Dashboard counters such as total scanned, files scanned, and columns scanned.
privacera_lineage Source-to-destination lineage events, movement between data zones, and related user/event attributes. Data Zone Movement; resource lineage graph; Events user filter.
privacera_alerts Policy and compliance alert events, including severity, policy name, and related resource or zone context. Alerts Dashboard; alert highlighting on Data Zone Movement.
privacera_resource_meta_info Resource type and location metadata (for example, Hive metastore path information) used to refine classification exports. Used during classification report export to skip warehouse file paths. No dedicated menu.
privacera_logs Operational and activity logs from Discovery components (message, level, host, timestamp; may include stack traces). Settings → Logs.
privacera_metrics Operational metrics: timers, counters, gauges, meters, and histograms. Settings → Metrics and Metrics Charts.
privacera_spark_events Spark scan listener and processing events used for scan diagnostics. Scan diagnostics and diagnostic event export (Health Check and diagnostics).
privacera_offline_scan_summary Per-resource offline scan outcomes: tagged, untagged, excluded, failed, deleted, listing failed, and rescan metadata. Scan Status drill-down and scan-summary report export.
privacera_offline_scan_cleanup Temporary cleanup working-set entries (scan ID and resource name) used by cleanup jobs. Not a user-facing page. Retention here acts as a safety net; the application already deletes these records once cleanup finishes.
privacera_audits Access and related audit events used by the Portal Events view. Settings → Events (Events tab).
privacera_discovery_telemetry Aggregated Discovery telemetry: tag counts, scanned data volume, and data-source counts. Telemetry page (Discovery section).