Encryption - Databricks Unity Catalog¶

This guide provides instructions for setting up encryption support for Databricks Unity Catalog using Privacera Encryption Gateway (PEG).

Overview¶

With Privacera Encryption Gateway (PEG), you can encrypt sensitive data in Databricks Unity Catalog to enhance data security and meet regulatory compliance requirements.

Runtime processing¶

PEG applies protect, unprotect, and mask operations exclusively during live request processing.

All cryptographic operations execute in memory within the scope of each request — no data is written to disk, databases, or object storage at any point during the encryption process.

Plaintext, ciphertext, and key material are held in memory only for the duration needed to process the request and generate a response. Once the request completes, all such data is immediately discarded and never persisted.

As a result, PEG does not retain user data or request payloads between requests. The gateway is fully stateless — each request is processed independently, with no carry-over of state or data from prior requests.

To know general Privacera Encryption concepts, see Privacera Encryption.

Deployment options¶

Privacera offers three deployment types. The following sections describe each option and how protect, unprotect, and mask operations flow between your environment and Privacera.

Privacera Cloud¶

All encryption components run in Privacera's cloud. Privacera manages PEG, Scheme Server, metadata DB, and key management (AWS Secrets Manager). You run the client application (e.g., Databricks), which connects to Privacera for protect/unprotect operations.

sequenceDiagram
    box Privacera Cloud Environment
        participant APP as Client Application (DBX)
        participant PEG as PEG Server
        participant SS as Scheme Server
        participant DB as Metadata DB
        participant ASM as Privacera's AWS Secrets Manager
    end

    APP->>PEG: Protect / Unprotect (JWT)
    PEG->>SS: Get Scheme

    SS->>DB: Fetch Scheme Metadata
    DB-->>SS: Scheme + Encrypted Data Key
    Note right of SS: Data key encrypted with Master Key

    SS->>ASM: Get Master Key
    ASM-->>SS: Master Key
    SS->>SS: Decrypt Data Key (in-memory only)

    SS-->>PEG: Scheme Metadata (no plaintext key)

    PEG->>PEG: Encrypt / Decrypt (runtime only)
    PEG-->>APP: Encrypted / Decrypted Data

D2P (Privacera Cloud Data-plane)¶

D2P is Privacera Cloud Data-plane. PEG runs in your environment; Scheme Server, metadata DB, and key management run in Privacera's SaaS and cloud. Privacera manages Scheme Server, metadata DB, and key management (AWS Secrets Manager). You manage the client application and PEG in your own environment.

sequenceDiagram
    box Customer Environment
        participant APP as Client Application (DBX)
        participant PEG as PEG Server
    end

    box Privacera SaaS Environment
        participant SS as Scheme Server
        participant DB as Metadata DB
    end

    box Privacera Cloud (Key Management)
        participant ASM as Privacera's AWS Secrets Manager
    end

    APP->>PEG: Protect / Unprotect (JWT)
    PEG->>SS: Get Scheme

    SS->>DB: Fetch Scheme Metadata
    DB-->>SS: Scheme + Encrypted Data Key
    Note right of SS: Data key encrypted with Master Key

    SS->>ASM: Get Master Key
    ASM-->>SS: Master Key
    SS->>SS: Decrypt Data Key (in-memory only)

    SS-->>PEG: Scheme Metadata (no plaintext key)

    PEG->>PEG: Encrypt / Decrypt (runtime only)
    PEG-->>APP: Encrypted / Decrypted Data

Self-Managed¶

All components run in your environment with your own key management. Privacera provides the PEG and Scheme Server software. You manage PEG, Scheme Server, metadata DB, and your key management backend—typically Azure Key Vault (master key stays in the vault; unwrap returns plaintext DEK only).

sequenceDiagram
    box Customer Environment
        participant APP as Client Application (DBX)
        participant PEG as PEG Server
        participant SS as Scheme Server
        participant DB as Metadata DB
        participant KV as Azure Key Vault
    end

    APP->>PEG: Protect / Unprotect (JWT)
    PEG->>SS: Get Scheme

    SS->>DB: Fetch Scheme Metadata
    DB-->>SS: Scheme and Encrypted DEK
    Note right of DB: Encrypted DEK stored in DB

    SS->>KV: Decrypt encrypted scheme key
    Note right of KV: Operation in Key Vault, master key never exposed
    KV->>KV: Decrypt using master key
    KV-->>SS: Plaintext DEK

    SS-->>PEG: Scheme Metadata

    PEG->>PEG: Encrypt / Decrypt (runtime only)
    PEG-->>APP: Encrypted / Decrypted Data

Cluster Type Comparison¶