Send Audits to S3 in a Self-Managed AWS Deployment¶

Privacera Audit Server receives audits from different sources and uses Fluentd to send them to the configured destination. In a self-managed deployment, you can configure the Audit Server to send audit logs to Fluentd, which can then be configured to send the audits to external storage, such as Amazon Simple Storage Service (S3) on Amazon Web Services (AWS).

This is applicable only for self-managed deployments. For PrivaceraCloud, please reach out to your contact at Privacera.

Recommended on Amazon EKS

On Amazon EKS, bind an IAM role to the Audit Fluentd pod (IRSA) for S3 export so you can leave AUDIT_FLUENTD_S3_ACCESS_KEY and AUDIT_FLUENTD_S3_SECRET_KEY empty in vars.audit-fluentd.yml. See IAM for Audit Fluentd on Amazon EKS (IRSA).
If Audit Server itself calls AWS (for example SQS configured in vars.auditserver.yml), use IAM for Audit Server on Amazon EKS (IRSA) instead.

AWS authentication¶

Use an IAM identity for Fluentd instead of long-lived access keys whenever possible.

Deployment	Typical approach
Amazon EKS (recommended)	IRSA for Audit Fluentd: one IAM role on the Fluentd `ServiceAccount` allows S3 for audit export. Leave `AUDIT_FLUENTD_S3_ACCESS_KEY` and `AUDIT_FLUENTD_S3_SECRET_KEY` empty.
EC2 / instance profile	Attach an instance role with `s3:PutObject` (and prefix-scoped ARNs you need). Leave the S3 access keys empty.
Static keys (fallback)	Set `AUDIT_FLUENTD_S3_ACCESS_KEY` and `AUDIT_FLUENTD_S3_SECRET_KEY` only when you cannot use IAM.

Prerequisites¶

Prerequisite	Description
AWS account	Ability to use S3 and to grant Fluentd `s3:PutObject` (and related actions) on your bucket and prefix via IAM or access keys.
Bucket in S3	A bucket where audit objects are stored (for example `privacera_audits`).
Audit Server	Audit Server must be enabled (default in self-managed deployments).

Setup¶

To setup sending audit logs to S3, follow these steps:

SSH into the instance where Privacera Manager is installed.
Navigate to the config directory with the following command:
Bash
1
cd ~/privacera/privacera-manager/config
Run the following command to copy the file from sample vars. The -n flag ensures that the file is not overwritten if it already exists.
Bash
1
cp -n sample-vars/vars.audit-fluentd.yml custom-vars/
Run the following command to open the .yml file for editing.
Bash
1
vi custom-vars/vars.audit-fluentd.yml

Modify the following properties:

Property	Description
AUDIT_FLUENTD_AUDIT_DESTINATION	Set to `s3` to write audit objects to Amazon S3. Fluentd uses one cloud destination at a time. To send PolicySync audits to Amazon SQS, configure Audit Server instead; see Audits to SQS.
AUDIT_FLUENTD_S3_ACCESS_KEY	Optional. Leave empty when using IAM (instance profile, or EKS IRSA for Fluentd). Otherwise, the AWS access key for S3.
AUDIT_FLUENTD_S3_SECRET_KEY	Optional. The secret key paired with `AUDIT_FLUENTD_S3_ACCESS_KEY` when using static credentials.
AUDIT_FLUENTD_S3_BUCKET	The name of the S3 bucket where logs are to be stored.
AUDIT_FLUENTD_S3_PATH_PREFIX	The path prefix in the S3 bucket where logs will be stored.
AUDIT_FLUENTD_S3_REGION	The AWS region where the S3 bucket is located (default: `us-east-1`).

After configuring these properties you need to restart the services

Bash
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade 

Validation¶

To confirm that the setup is successful, after completing the steps above, perform an access operation to generate audit logs. Then, navigate to the S3 bucket and verify that the audit logs are stored in the specified path.

Tip

Note that the audits are not sent to S3 immediately. The Fluentd buffer is flushed based on the configured time interval. If you want to update the time interval, refer to the Advanced Configuration section.

Here is a sample S3 bucket Access audit logs:

S3 bucket Access audit

Advanced Configuration¶

Here are some of the advanced configurations that you can do to customize the audit logs sent to S3

Configure S3 Encryption

Configure S3 Encryption¶

For information on AWS S3 encryption, refer to the AWS documentation.

Prerequisites¶

Prerequisite	Description
S3 Bucket Encryption	Your S3 bucket must be encrypted.
Encryption Key	You must have an encryption key for server-side encryption.

Configuration¶

If your S3 bucket is encrypted, you can configure the encryption key in the vars.audit-fluentd.yml file.

SSH into the instance where Privacera Manager is installed.

Run the following command to open the .yml file for editing:

Bash
cd ~/privacera/privacera-manager/
vi config/custom-vars/vars.audit-fluentd.yml

To enable encryption for S3, include the following property in the vars.audit-fluentd.yml file:

Property	Description
AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_TYPE	Specifies the encryption type for the S3 bucket (e.g., `SSE-S3`, `SSE-KMS`, `SSE-C`, or `NONE`).
AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_KEY	Specifies the encryption key for `SSE-KMS` or `SSE-C` encryption types (required if `SSE-KMS` or `SSE-C` is used as encryption type).
AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_KEY_MD5	Specifies the MD5 hash of the encryption key for `SSE-C` encryption type. (required if `SSE-C` is used as encryption type).

Once the properties are configured, update your Privacera Manager platform instance by following these steps
Bash
1 2 3
cd ~/privacera/privacera-manager ./privacera-manager.sh setup ./pm_with_helm.sh upgrade

Configure Time Interval for Sending Audits to S3

Configure Time Interval for Sending Audits to S3¶

If you want to change default time intervals for sending audits to S3, you can configure the following properties in the vars.audit-fluentd.yml file

SSH into the instance where Privacera Manager is installed.

Run the following command to open the .yml file for editing:

Bash
cd ~/privacera/privacera-manager/
vi config/custom-vars/vars.audit-fluentd.yml

Modify the following properties:

Variable	Definition
AUDIT_FLUENTD_S3_FILE_TIME_INTERVAL	This parameter defines the time-based interval for Fluentd to create a new log file. The log file will be rotated or split based on this time interval.
AUDIT_FLUENTD_S3_FILE_TIME_WAIT	Allows Fluentd to wait for a specified duration after the timekey(`AUDIT_FLUENTD_S3_FILE_TIME_INTERVAL`) interval to ensure all logs for that time period are written before rotating the log file.
AUDIT_FLUENTD_S3_FLUSH_INTERVAL	Specifies how frequently Fluentd should push the audits to the S3 destination. Example: `30s`.

Once the properties are configured, update your Privacera Manager platform instance by following these steps
Bash
1 2 3
cd ~/privacera/privacera-manager ./privacera-manager.sh setup ./pm_with_helm.sh upgrade

Fluentd Payload and Buffer Configuration

Fluentd Payload and Buffer Configuration¶

If you want to change the default payload size and buffer size for Fluentd, you can configure the following properties in the vars.audit-fluentd.yml file

SSH into the instance where Privacera Manager is installed.

Run the following command to open the .yml file for editing:

Bash
cd ~/privacera/privacera-manager/
vi config/custom-vars/vars.audit-fluentd.yml   

Modify the following properties:

Property	Definition	Default
AUDIT_FLUENTD_BODY_SIZE_LIMIT_MB	This property sets the body_size_limit, which is the maximum size for incoming request bodies	100
AUDIT_FLUENTD_S3_FILE_SIZE	This property sets the chunk_limit_size, which is the maximum size of a data chunk created in fluentd	100m

Once the properties are configured, update your Privacera Manager platform instance by following these steps:
Bash
1 2 3
cd ~/privacera/privacera-manager ./privacera-manager.sh setup ./pm_with_helm.sh upgrade

Prev Advanced Configuration