Skip to main content

Privacera Documentation

Connect Kafka datasource via plugin to Privacera Platform

These are details for installing Privacera plugin software into your Kafka implementation when you want to protect your Kafka data via access management policies. (Privacera Platform has its own Kafka component, sometimes called "PKafka", to distribute audit data. That is a different topic.)

Privacera's plugin supports both open-source Apache Kafka and Confluent Kafka. Some settings during installation are different between them, as noted in the procedures that follow.

Prerequisites

Be sure you are ready with the following:

  • PrivaceraPlatform has been installed.

  • The domain name or IP address of your system where Kafka is installed.

  • Shell access to that system by a user with sufficient privileges to install software.

  • Directory name where Kafka is installed on that system.

  • Network path and name of zipfile of Privacera plugin for Kafka.

  • Hostnames and ports for Privacera's Apache Ranger and Solr services

  • Decide the name of the Privacera policy repository where you will create access management policies for your Kafka data.

Obtain name of and path to Privacera Kafka plugin for download

Contact your Privacera representative to get the network path and name of the zipfile of the Privacera plugin for Kafka for you to download and install. This is an example:

https://s3.amazonaws.com/privacera/ranger/ranger-1.2.1-SNAPSHOT-kafka-plugin.tar.gz

In the procedures here, this path and zipfile name are shown as:

https://s3.amazonaws.com/privacera/ranger/<PRIVACERA_KAFKA_PLUGIN>.tar.gz

Determine hostnames and ports for Privacera's Apache Ranger and Solr services

When you installed Privacera Platform, you might have installed different Privacera Platform services on different host pods and changed the default ports for those services.

Installing the Privacera plugin for Kafka requires that you know the hostnames and ports for the following Privacera services.

  • Apache Ranger

  • Solr

The default port numbers for these services are listed in Default Privacera Platform port numbers.

Check with the people who installed Privacera Platform to determine the hostnames and ports for these services.

Name of Privacera policy repository

Privacera access management policies are stored in a policy repository. During installation, you specify the name of the repository, which can be anything you want. However, for clarity, Privacera recommends this convention:

privacera_kafka

Procedure to install Kafka plugin

The general sequence of steps to install the Privacera Kafka plugin is as follows:

  • Download and unzip the software

  • Set configuration parameters

  • Create and add an initialization script to Kafka

You can install the Privacera Kafka plugin software anywhere you prefer on your Kafka system. In the following procedure, the software is downloaded to /opt/privacera/downloads and installed in /opt/privacera/kafka-plugin.

  1. Download and unzip the software.

    # Make a directory to hold the download and change to it
    mkdir -p /opt/privacera/downloads
    cd /opt/privacera/downloads
    # Get the zipfile of the plugin
    wget https://s3.amazonaws.com/privacera/ranger/<PRIVACERA_KAFKA_PLUGIN>.tar.gz
    # Untar the zipfile
    cd /opt/privacera/
    tar xfz /opt/privacera/downloads/<PRIVACERA_KAFKA_PLUGIN>.tar.gz
    # Make a softlink to the unzipped directory
    ln -sf /opt/privacera/<PRIVACERA_KAFKA_PLUGIN> kafka-plugin         
  2. Set variables in the install.properties file.

    cd /opt/privacera/kafka-plugin/
    mkdir -p /opt/privacera/audit/solr/spool
    # Set values in install.properties file
    vi install.properties 
    COMPONENT_INSTALL_DIR_NAME=${KAFKA_CONF_DIR}
    POLICY_MGR_URL=http://${RANGER_HOST}:${RANGER_PORT}
    REPOSITORY_NAME=${PRIVACERA_POLICY_REPOSITORY_NAME}
    XAAUDIT.SOLR.ENABLE=true
    XAAUDIT.SOLR.URL=http://${SOLR_HOSTNAME}:${SOLR_PORT}/solr/ranger_audits
    XAAUDIT.SOLR.FILE_SPOOL_DIR=/opt/privacera/audit/solr/spool
    XAAUDIT.SOLR.IS_ENABLED=true
    XAAUDIT.SOLR.SOLR_URL=http://${SOLR_HOSTNAME}:${SOLR_PORT}/solr/ranger_audits  
  3. Enable the Privacera plugin for Kafka with enable-kafka-plugin.sh.

    vi enable-kafka-plugin.sh 
    #
    # Go to line 226 and remove config from end
    #
    # Set Java home directory
    export JAVA_HOME=`jrunscript -e 'java.lang.System.out.println(java.lang.System.getProperty("java.home"));'`
    # Run the script
    ./enable-kafka-plugin.sh    
  4. Create plugin initialization script kafka-ranger-env.sh and add it to Kafka startup configuration.

    # Create kafka-ranger-env.sh initialization script
    vi ${KAFKA_CONF_DIR}/kafka-ranger-env.sh
    classpathmunge () {
            escaped=`echo $1 | sed -e 's:\*:\\\\*:g'`
            if ! echo ${CLASSPATH} | /bin/egrep -q "(^|:)${escaped}($|:)" ; then
               if [ "$2" = "before" ] ; then
                  CLASSPATH=$1:${CLASSPATH}
               else
                  CLASSPATH=${CLASSPATH}:$1
               fi
            fi
    }
    classpathmunge '/usr/lib/hadoop-hdfs/lib/*'
    #
    # NOTE: the following three lines are
    # specific to installing the Privacera plugin
    # on Confluent Kafka.
    #
    # If you are installing the Privacera plugin
    # on open-source Apache Kafka, comment or do not add
    # these lines.
    #
    classpathmunge '${KAFKA_CONF_DIR}'     
    classpathmunge '${KAFKA_CONF_DIR}/libs/*'
    classpathmunge '${KAFKA_CONF_DIR}/libs/ranger-kafka-plugin-impl/*'
    #
    # END Confluent-Kafka-specific settings
    #
    export CLASSPATH
    unset classpathmunge
    
    chmod +x ${KAFKA_CONF_DIR}/kafka-ranger-env.sh
    
    # Add this in Kafka's kafka-run-class script
    vi ${KAFKA_HOME}/bin/kafka-run-class
    # Go to line 311 and enter the following line
    # to run the Privacera plugin when Kafka starts
    . ${KAFKA_CONF_DIR}/kafka-ranger-env.sh
    
  5. Restart Kafka as you normally would.

Test the Kafka plugin

To test Privacera's plugin for Kafka, you need users and policies defined in Privacera, which are extensive topics separate from installing the plugin. See: