AuditServer with Kafka
This topic covers how you can configure Kafka audit endpoint in AuditServer for the Ranger plugin and the Ranger Admin to send the audits.
Prerequisites
Ensure the following prerequisites are met:
- AuditServer must be configured. For more information, click here.
CLI Configuration
-
SSH to an instance where Privacera is installed.
-
Run the following commands.
cd ~/privacera/privacera-manager cp config/sample-vars/vars.auditserver.kafka.destination.yml config/custom-vars/ vi config/custom-vars/vars.auditserver.kafka.destination.yml
-
Modify the properties. For property details and description, refer to the Configuration Properties below.
AUDITSERVER_KAFKA_DESTINATION: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_BROKER_LIST: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_TOPIC_NAME: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SECURITY_PROTOCOL: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SSL_KEY_PASSWORD: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SASL_JAAS_CONFIG: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SASL_MECHANISM: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_OAUTH_WITH_SSL: " <PLEASE_CHANGE>" AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER: "<PLEASE_CHANGE>" AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_OAUTH_CLIENT_ID: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET: "<PLEASE_CHANGE>" AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR: "/workdir/privacera-audit-server/kafka-spool" ADMIN_AUDITSERVER_KAFKA_DESTINATION: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_BROKER_LIST: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_TOPIC_NAME: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SECURITY_PROTOCOL: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SSL_KEY_PASSWORD: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SASL_JAAS_CONFIG: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SASL_MECHANISM: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_OAUTH_WITH_SSL: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_ID: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET: "<PLEASE_CHANGE>" ADMIN_AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR: "/workdir/privacera-audit-server/kafka-spool"
-
Run the following commands.
cd ~/privacera/privacera-manager ./privacera-manager.sh update
Configuration Properties
The property names prefixed with ADMIN_ refer to Privacera Ranger Admin, whereas the others refer to Privacera Portal.
Property | Description | Example |
---|---|---|
AUDITSERVER_KAFKA_DESTINATION ADMIN_AUDITSERVER_KAFKA_DESTINATION |
Set to true if audit destination is kafka | |
AUDITSERVER_KAFKA_BROKER_LIST ADMIN_AUDITSERVER_KAFKA_BROKER_LIST |
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down). | 10.xxx.xx.xxx:9093 |
AUDITSERVER_KAFKA_TOPIC_NAME ADMIN_AUDITSERVER_KAFKA_TOPIC_NAME |
Topic name to which audits are to be sent | topic-name |
AUDITSERVER_KAFKA_SECURITY_PROTOCOL ADMIN_AUDITSERVER_KAFKA_SECURITY_PROTOCOL |
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. |
SASL_SSL |
AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION |
The location of the key store file. Make sure key is copied in config/ssl folder. Provide name of the file. |
kafka.server.keystore |
AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD |
The store password for the key store file.This is optional and only needed if AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION is configured. | privacera |
AUDITSERVER_KAFKA_SSL_KEY_PASSWORD ADMIN_AUDITSERVER_KAFKA_SSL_KEY_PASSWORD |
The password of the private key in the key store file. This is optional. | privacera |
AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION |
The location of the trust store file. Make sure the key is copied in config/ssl folder. Provide name of the file. | kafka.server.truststore |
AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD |
The password for the trust store file. | privacera |
AUDITSERVER_KAFKA_SASL_JAAS_CONFIG ADMIN_AUDITSERVER_KAFKA_SASL_JAAS_CONFIG |
Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration. You must provide JAAS configurations for all SASL authentication mechanisms. E.g "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required ;" if AUDITSERVER_KAFKA_SASL_MECHANISM is "OAUTHBEARER |
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required ; |
AUDITSERVER_KAFKA_SASL_MECHANISM ADMIN_AUDITSERVER_KAFKA_SASL_MECHANISM |
SASL mechanism used for connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism. | OAUTHBEARER |
AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS ADMIN_AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS |
The LoginModule for the selected SASL_MECHANISM E.g "io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler" if AUDITSERVER_KAFKA_SASL_MECHANISM is "OAUTHBEARER |
io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler |
AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI ADMIN_AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI |
OAUTH Token endpoint URL used by the application in order to get an access token or a refresh token | http://10.211.93.140:4444/oauth2/token |
AUDITSERVER_KAFKA_OAUTH_WITH_SSL ADMIN_AUDITSERVER_KAFKA_OAUTH_WITH_SSL |
Set to true if SSL is applied on OAUTH. | |
AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER ADMIN_AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER |
Set to true if OAUTH accept unsecure server. | |
AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE ADMIN_AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE |
The authorization server needs to know which grant type the application wants to use since it affects the kind of credential it will issue e.g client_credentials |
client_credentials |
AUDITSERVER_KAFKA_OAUTH_CLIENT_ID ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_ID |
The ID of the application that asks for authorization. | broker-kafka |
AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET |
The secret of the application that asks for authorization. | broker-kafka |
AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR ADMIN_AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR |
If audit framework detects that an audit destination is down then it buffers the audit messages in memory. Once memory buffer fills up then it can be configured to spool the unsent messages to disk files to prevent or minimize the loss of audit messages. Local disk directory where spool files would be kept. This value must be specified. Default location is "/workdir/privacera-audit-server/kafka-spool |
/workdir/privacera-audit-server/kafka-spool |