Skip to content

Omni Connector Event Ingestion

Event ingestion lets resource-change events from Amazon MSK or Apache Kafka update an Omni Connector in near real time, instead of waiting for the next scheduled scan. An AWS Lambda consumes a topic, maps each event, and posts it to the connector's /events endpoint. The connector then reads the changed object from the platform and updates its inventory.

Event ingestion is available for Databricks Unity Catalog, Databricks SQL and AWS Lake Formation connectors. It is off by default, and nothing is deployed until you download and apply a Lambda setup bundle.

How it is configured

Configuration is in two places. Configure the plane first, then each connector.

Where What you set
Runtime Plane: the OmniConnector system application, under System Applications Whether event ingestion is on, the event source, broker or cluster settings, the AWS account, VPC, subnets, security group and staging bucket, the connector endpoint, and Event API auth.
Each connector: the Event Ingestion section of the connector form Whether this connector takes part, the event source, topic, source names and resource prefix.

Custom properties

Plane-wide event ingestion settings are set once, in the OmniConnector system application. On a Shared connector, a custom property whose name starts with event-ingestion. is not accepted. A Dedicated connector may set one for its own pod, except Event API auth, the JWT token and the staging bucket, which only the plane sets.

Before you begin

  • An Omni Connector of a supported platform, running on a Self-Managed Runtime Plane.
  • An Amazon MSK cluster or Apache Kafka brokers that the Lambda's VPC can reach.
  • An AWS account and a VPC with subnets and a security group for the Lambda.
  • An S3 staging bucket in the Runtime Plane's own region. AWS does not read Lambda code across regions, and the Privacera release bucket is in us-east-1.
  • A connector endpoint that the Lambda's VPC can reach. Cluster-local DNS names normally cannot be reached, so use an internal load balancer or ingress address.

D2P

Download Lambda setup is refused on Partially Managed (D2P) planes. D2P has no supported credential path for downloading the artifacts.

Configure the Runtime Plane

In the OmniConnector system application, open the Event Ingestion section and set:

Field Description
Enable event ingestion Turns on Lambda-based event ingestion for the connectors on this plane.
Event source Amazon MSK or Apache Kafka. Each source has its own Lambda package, CloudFormation template and settings. Connectors that take part must use the same source.
AWS account ID, VPC, Subnets, Security group Where the Lambda runs. The VPC must reach both the event source and the connector endpoint.
Lambda staging bucket The bucket the Lambda package is uploaded to. Must be in the Runtime Plane's region.
Connector endpoint The base URL the Lambda posts events to. It is pre-filled from the connector's ingress.
Event API auth None, or JWT. With JWT, the connector verifies the Lambda's token against the plane's JWT signing keys and requires the omni-connector scope. JWT is available on Self-Managed planes only. It is required for Lake Formation federation.

For Amazon MSK, also set the MSK cluster name, MSK cluster UUID and MSK starting position (Latest skips the existing backlog, Trim horizon reads from the start).

For Apache Kafka, also set:

Field Description
Kafka bootstrap servers Comma-separated host:port pairs.
Kafka authentication Client certificate (TLS). It must match the secret named below.
Kafka Secrets Manager key ARN of the Secrets Manager secret used to authenticate to the brokers.
Server root CA secret ARN Optional. For brokers whose certificates are signed by a private or self-signed CA.
Kafka starting position, Kafka batch size, Kafka batch window (seconds) Where a new event source mapping starts reading (Latest or Trim horizon), the maximum records per invocation (default 100), and the maximum wait for a full batch (default 30).

Configure each connector

In the connector's Event Ingestion section:

  1. Turn on Enable event ingestion.
  2. Select the Event source. It must match the plane's source.
  3. Enter the topic: MSK topic, or Kafka topic and optionally Kafka consumer group ID. Use one topic per Lambda. Two connectors that share a topic process every event twice.
  4. Enter Source names: the comma-separated names of the sources that feed this connector, for example GDC_APP,UC_APP. Each becomes an entry in the routing configuration.
  5. Optionally set Resource prefix. It names the Lambda and its IAM role, and must be unique per Lambda in the AWS account. It defaults from the connector name.
  6. Optionally set Connector endpoint override if the generated ingress host is unavailable or the Lambda must use an internal load balancer.
  7. Review the generated Routing configuration. Edit it only if you need a shape the generator does not cover. After you edit it, Runtime Manager no longer maintains it automatically. The connectorId in it must match the connector shown in the portal.

Download and deploy the Lambda

  1. On the Runtime Plane, open System Applications, find the OmniConnector application, and select Download Lambda setup. The bundle covers every connector that has event ingestion on, with one CloudFormation stack per connector.
  2. Unzip it on a machine with Bash and the AWS CLI configured for the target account, then upload the Lambda package once and deploy:
    Bash
    1
2
3
4
    unzip omni-connector-lambda-setup.zip
cd omni-connector-lambda-setup
./deploy.sh upload
./deploy.sh
    ./deploy.sh lists the connector stacks and prompts for an action: create, update, reapply, status or delete. To run non-interactively:
    Bash
    1
2
3
    ./deploy.sh create 1
./deploy.sh reapply all
./deploy.sh status 2
    Stacks are independent, so changing one does not affect the others.

Treat the bundle as sensitive

The bundle can contain credentials, including a connector JWT. Store and share it like any sensitive configuration export.

The download is refused when:

  • event ingestion is off on the plane, or no connector has opted in;
  • a required plane or connector value is missing or invalid;
  • the enabled connectors use different event sources;
  • the plane is D2P;
  • Event API auth is JWT on a plane that is not Self-Managed.

If the bundle's README.md begins with a notice that it is incomplete, do not deploy it. Download it again once the artifact service can supply the missing files.

Key rotation invalidates the Lambda token

After any key rotation, redeploy the Lambda

With Event API auth set to JWT, the Lambda's token does not expire. However, any key rotation on the plane invalidates it, including a routine configuration-key rotation, which also replaces the JWT signing key.

The old key stops being accepted after the passive TTL (30 minutes by default). From then on, once the connector pods restart, every batch the Lambda sends is rejected with 401.

After any key rotation:

  1. Download the Lambda setup bundle again.
  2. Redeploy the Lambda.
  3. Restart the connector pods.

Runtime Manager logs a warning, "event-ingestion Lambda tokens stop verifying after the passive TTL", when this happens on a plane that uses JWT.

Troubleshooting

If the Lambda deploys but events do not arrive, check in this order:

  1. The Lambda has an event source mapping for the configured topic.
  2. The connector endpoint is reachable from the Lambda's VPC.
  3. Event API auth matches on both sides. It is one setting for both. With JWT, the bundle carries the Lambda's token and the connector requires the omni-connector scope. A 401 means the token is missing or not signed by this plane's key. A 403 means it lacks the scope. If you change the mode, the connector pods restart, so download the bundle again and redeploy the Lambda. See also Key rotation invalidates the Lambda token.
  4. The OmniConnectorUrl in the parameter file and the routing connectorId identify the intended connector. A wrong ID returns 404, and the Lambda fails the invocation instead of dropping events, so that partition stops advancing until it is fixed.
  5. The CloudFormation stack and the Lambda logs show no invocation or delivery errors.

If events are still rejected although the auth modes match, look for a custom property or environment variable that changes how the connectors verify events, such as connector.event-api-auth.*, spring.security.oauth2.resourceserver.* or EVENT_API_JWT_*. Shared connectors use the OmniConnector system application's settings, and a Dedicated connector uses its own. Runtime Manager does not manage or detect these.