AuditServer with Kafka#

This topic covers how you can configure Kafka audit endpoint in AuditServer for the Ranger plugin and the Ranger Admin to send the audits.

Prerequisites#

Ensure the following prerequisites are met:

AuditServer must be configured. For more information, click here.

UI Configuration#

Launch PM UI. For more information on how to start PM UI, click here.
In the left navigation of the PM UI, navigate to Setup Default Services > Configure Audit Server > Advanced.

Kafka Destination is displayed.

Click Kafka Destination to modify the properties. For property details and description, refer to the Configuration Properties below.
To install/update, go to Install in the left navigation, and click Install & Start Services.

CLI Configuration#

SSH to an instance where Privacera is installed.

Run the following commands.

cd ~/privacera/privacera-manager
cp config/sample-vars/vars.auditserver.kafka.destination.yml config/custom-vars/
vi config/custom-vars/vars.auditserver.kafka.destination.yml

Modify the properties. For property details and description, refer to the Configuration Properties below.

AUDITSERVER_KAFKA_DESTINATION: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_BROKER_LIST: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_TOPIC_NAME: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SECURITY_PROTOCOL: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SSL_KEY_PASSWORD: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SASL_JAAS_CONFIG: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SASL_MECHANISM: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_OAUTH_WITH_SSL: " <PLEASE_CHANGE>"
AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER: "<PLEASE_CHANGE>"
AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_OAUTH_CLIENT_ID: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET: "<PLEASE_CHANGE>"
AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR: "/workdir/privacera-audit-server/kafka-spool"
ADMIN_AUDITSERVER_KAFKA_DESTINATION: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_BROKER_LIST: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_TOPIC_NAME: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SECURITY_PROTOCOL: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SSL_KEY_PASSWORD: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SASL_JAAS_CONFIG: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SASL_MECHANISM: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_OAUTH_WITH_SSL: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_ID: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET: "<PLEASE_CHANGE>"
ADMIN_AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR: "/workdir/privacera-audit-server/kafka-spool"

Run the following commands.

cd ~/privacera/privacera-manager
./privacera-manager.sh update

Configuration Properties#

The property names prefixed with ADMIN_ refer to Privacera Ranger Admin, whereas the others refer to Privacera Portal.

Property	Description	Example
AUDITSERVER_KAFKA_DESTINATION ADMIN_AUDITSERVER_KAFKA_DESTINATION	Set to true if audit destination is kafka
AUDITSERVER_KAFKA_BROKER_LIST ADMIN_AUDITSERVER_KAFKA_BROKER_LIST	A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).	10.xxx.xx.xxx:9093
AUDITSERVER_KAFKA_TOPIC_NAME ADMIN_AUDITSERVER_KAFKA_TOPIC_NAME	Topic name to which audits are to be sent	topic-name
AUDITSERVER_KAFKA_SECURITY_PROTOCOL ADMIN_AUDITSERVER_KAFKA_SECURITY_PROTOCOL	Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.	SASL_SSL
AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION	The location of the key store file. Make sure key is copied in config/ssl folder. Provide name of the file.	kafka.server.keystore
AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD ADMIN_AUDITSERVER_KAFKA_SSL_KEYSTORE_PASSWORD	The store password for the key store file.This is optional and only needed if AUDITSERVER_KAFKA_SSL_KEYSTORE_LOCATION is configured.	privacera
AUDITSERVER_KAFKA_SSL_KEY_PASSWORD ADMIN_AUDITSERVER_KAFKA_SSL_KEY_PASSWORD	The password of the private key in the key store file. This is optional.	privacera
AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_LOCATION	The location of the trust store file. Make sure the key is copied in config/ssl folder. Provide name of the file.	kafka.server.truststore
AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD ADMIN_AUDITSERVER_KAFKA_SSL_TRUSTSTORE_PASSWORD	The password for the trust store file.	privacera
AUDITSERVER_KAFKA_SASL_JAAS_CONFIG ADMIN_AUDITSERVER_KAFKA_SASL_JAAS_CONFIG	Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration. You must provide JAAS configurations for all SASL authentication mechanisms. E.g "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required ;" if AUDITSERVER_KAFKA_SASL_MECHANISM is "OAUTHBEARER	org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required ;
AUDITSERVER_KAFKA_SASL_MECHANISM ADMIN_AUDITSERVER_KAFKA_SASL_MECHANISM	SASL mechanism used for connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.	OAUTHBEARER
AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS ADMIN_AUDITSERVER_KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS	The LoginModule for the selected SASL_MECHANISM E.g "io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler" if AUDITSERVER_KAFKA_SASL_MECHANISM is "OAUTHBEARER	io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler
AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI ADMIN_AUDITSERVER_KAFKA_OAUTH_TOKEN_ENDPOINT_URI	OAUTH Token endpoint URL used by the application in order to get an access token or a refresh token	http://10.211.93.140:4444/oauth2/token
AUDITSERVER_KAFKA_OAUTH_WITH_SSL ADMIN_AUDITSERVER_KAFKA_OAUTH_WITH_SSL	Set to true if SSL is applied on OAUTH.
AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER ADMIN_AUDITSERVER_OAUTH_ACCEPT_UNSECURE_SERVER	Set to true if OAUTH accept unsecure server.
AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE ADMIN_AUDITSERVER_OAUTH_LOGIN_GRANT_TYPE	The authorization server needs to know which grant type the application wants to use since it affects the kind of credential it will issue e.g client_credentials	client_credentials
AUDITSERVER_KAFKA_OAUTH_CLIENT_ID ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_ID	The ID of the application that asks for authorization.	broker-kafka
AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET ADMIN_AUDITSERVER_KAFKA_OAUTH_CLIENT_SECRET	The secret of the application that asks for authorization.	broker-kafka
AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR ADMIN_AUDITSERVER_KAFKA_BATCH_FILESPOOL_DIR	If audit framework detects that an audit destination is down then it buffers the audit messages in memory. Once memory buffer fills up then it can be configured to spool the unsent messages to disk files to prevent or minimize the loss of audit messages. Local disk directory where spool files would be kept. This value must be specified. Default location is "/workdir/privacera-audit-server/kafka-spool	/workdir/privacera-audit-server/kafka-spool