Skip to content

Dremio

This section covers how you can integrate Dremio with Privacera. You can use Dremio for table-level access control with native Ranger plugin.

By integrating Dremio with Privacera, you'll be provided with comprehensive data lake security and fine-grained access control across multi-cloud environments. Dremio works directly with data lake storage. Using Dremio's query engine and ability to democratize data access, Privacera implements fine-grained access control policies, then automatically enforces and audits them at enterprise scale.

Dremio is supported with the following data sources:

  • S3
  • ADLS
  • Hive
  • Redshift

Prerequisite

Ensure the following prerequisite is met:

  • A Privacera Manager host where Privacera services are running.

  • A Dremio host where Dremio Enterprise Edition is installed. (Community Edition is not supported)

Configuration

  1. Run the following commands.

    cd ~/privacera/privacera-manager
    cp config/sample-vars/vars.dremio.yml config/custom-vars/
    
  2. Run the following commands.

    cd ~/privacera/privacera-manager
    ./privacera-manager.sh update
    

    After the update is completed, the Dremio plugin installation script privacera_dremio.sh and custom configuration archive privacera_custom_conf.tar.gz is generated at the location ~/privacera/privacera-manager/output/dremio.

  3. Configure Privacera plugin depending on how you have installed Dremio in your instance.

    Note

    For a new/existing data source configured in Dremio Data Lake, ensure Enable external authorization plugin checkbox under Settings > Advanced Options of the data source is selected in the Dremio UI. Then, restart the Dremio service.

    Depending on your cloud provider, set up Dremio in a Kubernetes environment. See the following links:

    After setting up Dremio, perform the following steps to deploy Privacera plugin. The steps assume that your Privacera Manager host instance is separate from your Dremio Kubernetes instance. If they are configured on the single instance, then modify the steps accordingly.

    1. SSH to your instance where Dremio is installed containing the Dremio Kubernetes artifacts and change to the dremio-cloud-tools/charts/dremio_v2/ directory.

    2. Copy the privacera_dremio.sh and privacera_custom_conf.tar.gz files from your Privacera Manager host instance to the dremio_v2 folder in your Dremio Kubernetes instance.

    3. Run the following commands:

      mkdir -p privacera_config
      mv privacera_dremio.sh privacera_config/
      mv privacera_custom_conf.tar.gz privacera_config/
      
    4. Update configmap.yml to add new configmap for Privacera configuration.

      vi templates/dremio-configmap.yaml
      

      Add the following configuration at the start of the file:

      apiVersion: v1
      kind: ConfigMap
      metadata:
      name: dremio-privacera-install
      data:
      privacera_dremio.sh: |- {{ .Files.Get "privacera_config/privacera_dremio.sh" | nindent 4 }}
      binaryData:
      privacera_custom_conf.tar.gz: {{ .Files.Get "privacera_config/privacera_custom_conf.tar.gz" | b64enc | nindent 4 }}
      ---
      
    5. Update dremio-env to add Privacera jars and configuration in the Dremio classpath.

      vi config/dremio-env
      

      Update the following variable if it exists or add it.

      DREMIO_EXTRA_CLASSPATH=/opt/privacera/conf:/opt/privacera/dremio-ext-jars/*
      
    6. Update values.yaml.

      vi values.yaml
      

      Add the following configuration for extraInitContainers inside the coordinator section.

      extraInitContainers:  |
          - name: install-privacera-dremio-plugin
          image: {{.Values.image}}:{{.Values.imageTag}}
          imagePullPolicy: IfNotPresent
          securityContext:
              runAsUser: 0
          volumeMounts:
          - name: dremio-privacera-plugin-volume
              mountPath: /opt/dremio/plugins/authorizer
          - name: dremio-ext-jars-volume
              mountPath: /opt/privacera/dremio-ext-jars
          - name: dremio-privacera-config
              mountPath: /opt/privacera/conf/
          - name: dremio-privacera-install
              mountPath: /opt/privacera/install/
          command:
              - "bash"
              - "-c"
              - "cd /opt/privacera/install/ && cp * /tmp/ && cd /tmp && ./privacera_dremio.sh"
      

      Update or uncomment the extraVolumes section inside the coordinator section and add the following configuration:

      extraVolumes:
      - name: dremio-privacera-install
          configMap:
          name: dremio-privacera-install
          defaultMode: 0777
      - name: dremio-privacera-plugin-volume
          emptyDir: {}
      - name: dremio-ext-jars-volume
          emptyDir: {}
      - name: dremio-privacera-config
          emptyDir: {}
      

      Update or uncomment the extraVolumeMounts section inside the coordinator section and add the following configuration:

      extraVolumeMounts:
      - name: dremio-ext-jars-volume
          mountPath: /opt/privacera/dremio-ext-jars
      - name: dremio-privacera-plugin-volume
          mountPath: /opt/dremio/plugins/authorizer
      - name: dremio-privacera-config
          mountPath: /opt/privacera/conf
      
    7. Upgrade your Helm release. Get the release name by running helm list command. The text under the Name column is your Helm release.

      helm upgrade -f values.yaml <release-name>
      
    1. SSH to your instance where Dremio RPM is installed.

    2. Copy the privacera_dremio.sh and privacera_custom_conf.tar.gz files from your Privacera Manager host instance to the Home folder in your Dremio instance.

    3. Run the following commands:

      mkdir -p ~/privacera/install
      mv privacera_dremio.sh ~/privacera/install
      mv privacera_custom_conf.tar.gz ~/privacera/install
      
    4. Launch the privacera_dremio.sh script.

      cd ~/privacera/install
      chmod +x privacera_dremio.sh
      sudo ./privacera_dremio.sh
      
    5. Update dremio-env to add Privacera jars and configuration in the Dremio classpath.

      vi ${DREMIO_HOME}/conf/dremio-env
      

      Update the following variable if it exists or add it.

      DREMIO_EXTRA_CLASSPATH=/opt/privacera/conf:/opt/privacera/dremio-ext-jars/*
      
    6. Restart Dremio.

      sudo service dremio restart