Skip to content

Send Audits to Amazon SQS in a Self-Managed AWS Deployment

In a self-managed deployment, Audit Server can send PolicySync audits to Amazon SQS using the AWS SDK. Enable the SQS destination and provide the queue details in Privacera Manager; Audit Server uses your AWS credentials (for example an IAM role on the cluster) to publish messages.

For audits written to object storage through Audit Fluentd, see Send Audits to S3. To send the same connector audit stream to Solr, Audit Fluentd, and Amazon SQS together, see Connector audit output destinations.

This is applicable only for self-managed deployments. For PrivaceraCloud, please reach out to your contact at Privacera.

Note

Note

Use this page when you want Privacera connector activity—for example grant and revoke operations the connector runs—to show up as messages in your Amazon SQS queue. After you follow the setup below, Audit Server publishes those connector events to SQS. Audit traffic that does not go through a connector (for example some Ranger-only flows) is not covered here.

Collections supported for SQS

Audit stream Audit collection SQS on Audit Server
PolicySync audits policysync_audit Supported. Complete the Setup section in this guide.

Prerequisites

Requirement Notes
Traffic policysync_audit events from connectors (including grant and revoke actions executed by a connector).
Queue Standard or FIFO queue URL and matching region.
IAM sqs:SendMessage on the Audit Server identity (plus GetQueueUrl / GetQueueAttributes if you use them).
Handler V2 Connector audit handler V2 must be on. Set AUDITSERVER_POLICYSYNC_AUDIT_HANDLER_V2_ENABLED to true as described in Connector audit output destinations.

Setup

To send PolicySync audits to Amazon SQS, follow these steps.

  1. Sign in to the host where Privacera Manager is installed.
  2. Open the Manager configuration directory:
    Bash
    1
    cd ~/privacera/privacera-manager/config
  3. Copy the Audit Server sample variables into custom-vars if you have not already. The -n flag avoids overwriting an existing file.
    Bash
    1
    cp -n sample-vars/vars.auditserver.yml custom-vars/
  4. Edit the Audit Server custom variables file:
    Bash
    1
    vi custom-vars/vars.auditserver.yml
  5. Turn on connector audit handler V2 and set the destination list as described in Connector audit output destinations. When Amazon SQS is enabled for this stream, the destination list must include sqs (Privacera Manager does not add it automatically).

  6. In the same file, set the Amazon SQS properties:

    Property Purpose
    AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_ENABLED Set to true to send PolicySync audits to your queue.
    AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_QUEUE_URL Full HTTPS URL of the queue.
    AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_REGION AWS Region of the queue (for example us-east-1).
    AUDITSERVER_POLICYSYNC_AUDIT_DESTINATIONS_NAMES Comma-separated destinations; include sqs when SQS is on (for example solr,sqs). See Connector audit output destinations for solr, fluentd, and sqs.

    If Amazon SQS delivery fails temporarily, Audit Server can spool messages and retry. In the same file, optional SQS spool settings are:

    Property Purpose
    AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_SPOOL_ENABLED Set to true to write failed SQS sends to the SQS spool under the shared PolicySync spool base directory.
    AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_SPOOL_DRAIN_INTERVAL_SECONDS How often the SQS spool is drained and retried, in seconds.

    The base directory and drain interval shared by Solr, SQS, and Fluentd for connector audits are described in Connector audit output destinations. For Fluentd URL and path settings when fluentd appears in the destination list, use the Audit Server sample variables and the same connector destinations guide.

  7. Apply the configuration with Privacera Manager, then restart Audit Server.

Bash
1
2
3
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade

Example

Solr plus one SQS queue (replace URL and region):

YAML
1
2
3
4
5
AUDITSERVER_POLICYSYNC_AUDIT_DESTINATIONS_NAMES: "solr,sqs"
AUDITSERVER_POLICYSYNC_AUDIT_HANDLER_V2_ENABLED: "true"
AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_ENABLED: "true"
AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_QUEUE_URL: "https://sqs.us-east-1.amazonaws.com/123456789012/policysync-audit"
AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_SQS_REGION: "us-east-1"

To also send the same stream to Audit Fluentd, add fluentd to the names list and set AUDITSERVER_POLICYSYNC_AUDIT_DESTINATION_FLUENTD_ENABLED to true (see sample vars for URL and path keys).

Validation

  1. Generate policysync_audit traffic, then confirm Solr (if solr is in the destination list) and new messages in the queue (AWS console, aws sqs receive-message, or a short-lived consumer).
  2. On your Audit Server dashboard (for example Grafana), confirm SQS-related metrics move with traffic instead of staying flat.
  3. On failures, check AUDITSERVER_POLICYSYNC_AUDIT_DESTINATIONS_SPOOL_BASE_DIR and Audit Server logs.

One SQS message is not always one audit JSON line (batching/spool before SendMessage).