Data Products

Overview¶

The Data Products feature provides a powerful way to create reusable, business-aligned views of data that abstract physical implementation details from underlying schemas or storage platforms. Data Products enable organizations to define logical data assets (e.g., Customer Data, Health Claims, Transaction Logs) that can be accessed consistently across different data platforms without requiring data copying or movement.

Unlike traditional datasets that are tightly coupled to specific physical tables, Data Products are designed to be platform-agnostic and policy-aware. This means you can define a single logical data product that can be accessed through different underlying platforms (e.g., accessing data in a data lake via Snowflake) while maintaining a consistent business view.

Key Benefits:

• Logical Abstraction: Separate business concepts from physical implementation, allowing data products to remain generic and reusable
• Platform Independence: Access the same logical data product across different underlying data platforms
• Policy-Driven Access: Leverage Privacera's policy engine to dynamically apply appropriate restrictions based on data attributes, usage context, and legal obligations
• No Data Movement: Access data in its original location without requiring data copying or replication
• Scalable Governance: Apply policies at the data product level rather than encoding constraints into the product definition itself

Rather than encoding constraints into the Data Product name or definition (e.g., "California_Customers"), the solution allows Data Products to remain generic. Privacera's policy engine then dynamically applies appropriate restrictions based on data attributes, usage context, and legal obligations. This approach provides flexibility and scalability while maintaining strong governance controls.

Understanding Data Products¶

Platform-Agnostic Design¶

Data Products provide a logical abstraction layer that separates business concepts from physical implementation. This design philosophy offers several advantages:

Generic Definition:

• Data Products remain generic and do not encode specific constraints or filters in their definition
• Business-aligned naming (e.g., "Customer Data") rather than constraint-specific names (e.g., "California_Customers")
• Flexible and reusable across different use cases and contexts

Policy-Driven Access:

• Access restrictions are applied dynamically through Privacera's policy engine
• Policies can adapt based on data attributes, usage context, and legal obligations
• No need to create multiple data products for different access scenarios

Cross-Platform Consistency:

• Access the same logical data product across different underlying platforms
• Example: Access customer data stored in a data lake through Snowflake, Databricks, or other platforms
• Maintains consistent business view regardless of the access method

No Data Movement:

• Access data in its original location without requiring data copying or replication
• Reduces storage costs and data synchronization complexity
• Ensures users always access the most current data

Policy Integration¶

Data Products work seamlessly with Privacera's policy engine to provide sophisticated, context-aware access control:

Dynamic Restrictions:

• Policies apply restrictions based on multiple factors:
• Data attributes (e.g., sensitivity level, data classification)
• Usage context (e.g., user role, location, time of access)
• Legal obligations (e.g., GDPR, HIPAA compliance requirements)

Context-Aware Access:

• Access control adapts to the user's context and requirements
• Different users may see different views of the same data product based on policies
• Policies can combine multiple conditions for fine-grained control

Attribute-Based Policies:

• Leverage data attributes (from tags or metadata) for sophisticated access control
• Use tag-based policies to apply restrictions automatically
• Combine resource-based and tag-based policies for comprehensive governance

Accessing Data Products¶

To access Data Products:

1. Navigate to Governance Hub in the left sidebar
2. Select Data Products

The Data Products interface displays all data products that have been created and are accessible to your account. You can view data products in various states: draft or published.

Data Products Interface¶

The Data Products interface provides a comprehensive view of all data products in your organization, enabling you to create, manage, and monitor logical data abstractions.

Data Products List¶

The main view displays all data products with key information:

• Name: The business-aligned name of the data product
• Description: A brief description of what the data product represents
• Status: Current state of the data product (Draft, Published)
• Tags: Tags assigned to the data product for classification and organization
• Assets Count: Number of assets (tables, columns etc) associated with the data product
• Last Modified: Timestamp of the most recent update
• Created By: The user who created the data product

Search and Filter¶

Use the search and filter capabilities to quickly find specific data products:

• Search: Enter keywords to search accros data product names
• Status Filter: Filter by data product status (Draft, Published)
• Created By Filter: Filter data products by the user who created them
• Sort Options: Sort by name, creation date, or last modified date

Creating a Data Product¶

Creating a data product is a two-step process that involves defining the product information and then adding assets. Follow these steps to create a new data product:

Step 1: Product Information¶

1. Navigate to Governance Hub → Products
2. Click the Create Product button
3. In the creation page, provide the following information:
4. What is the name of the product? (Required): Enter a descriptive, business-aligned name for the data product
   • Naming Rules: Product names must follow these requirements:
     • Allowed characters: Letters (A-Z, a-z), numbers (0-9), underscores (_), hyphens (-), and spaces
     • Use clear, meaningful names that reflect the business purpose (e.g., "Customer Data", "Health Claims", "Transaction Logs")
     • Avoid encoding constraints or filters in the name (e.g., use "Customer Data" instead of "California_Customers")
5. What is the purpose of this product?: (Optional) Provide a detailed description of what the data product represents, its business purpose, and intended use cases
6. Choose an action:
7. Cancel: Discard the creation and return to the data products list
8. Save as Draft: Save the product information and create the data product in draft state. You can add assets later.
9. Next: Proceed to the asset selection page

Step 2: Adding Assets¶

After providing the product information and clicking Next, you'll be taken to the asset selection page where you can add assets to your data product.

Asset Selection Interface:

The asset selection page provides a comprehensive view of all available assets:

• Data Catalog Explore view: The left panel displays a hierarchical view of all services and assets available in Omni
• Navigate through the hierarchy to browse services, databases, catalogs, schemas, and tables
• Expand and collapse nodes to explore the asset hierarchy
• Assets from all configured data sources are displayed in the view
• Asset Preview: The right panel shows a preview of the selected asset
• When you select an asset from the Data Catalog Explore view, its details are displayed in the preview panel
• Preview includes asset information such as name, path, type, and metadata
• Review the asset details before adding it to your data product

Adding Assets:

1. In the asset selection page, browse the Data Catalog Explore view to find the assets you want to add
2. Select the asset in the Data Catalog Explore view to see its preview on the right side
3. Review the asset details in the preview panel
4. Repeat the process to add multiple assets from different services

Asset Management on Selection Page:

• Added Assets Count: The interface displays the count of assets you've added
• Clear All: Click this button to remove all selected assets and start over
• Close Button: Each added asset has a close button (X) to remove it individually
• Asset List: View all added assets in a list format

Completing Asset Selection:

After selecting your assets, choose an action:

• Cancel: Discard all changes and return to the data products list
• Back: Return to the product information page to modify name or description
• Save as Draft: Save the data product with selected assets in draft state. The data product will not be active until published.
• Publish: Publish the data product immediately. The data product will come into effect and become active.

Removing Assets from a Data Product¶

You can remove assets from a data product when they are no longer needed or when the data product scope changes. Removing assets only removes the association with the data product; it does not delete the underlying physical resources.

Removing Assets:

1. Navigate to Governance Hub → Products
2. Click on the data product name/tab to open its details page
3. Click on the Assets tab
4. Select one or more assets from the assets list that you want to remove
5. Click the Remove Selected button to remove all selected assets

Important Notes:

• Removing assets does not delete the underlying physical resources (tables, columns, etc.)
• The assets remain available in the Data Catalog and can be added to other data products
• If the data product is published, removing assets may affect users who are currently accessing the data product
• Consider unpublishing the data product before making significant asset changes

Publishing a Data Product¶

Publishing a data product makes it active and available for use. Only published data products come into effect and become accessible to users. You can publish a data product either during creation (from the asset selection page) or later from the data product details page.

Publishing During Creation:

When creating a new data product, you can publish it directly from the asset selection page:

1. Complete Step 1 (Product Information) and click Next
2. Add assets in Step 2 (Asset Selection)
3. Click the Publish button
4. The system will:
5. Create a default tag for the data product
6. Create a Ranger role for the data product
7. Publish the data product, making it active and effective

Publishing an Existing Draft Data Product:

If you saved a data product as draft, you can publish it later:

1. Navigate to Governance Hub → Products
2. Click on the data product name/tab to open its details page
3. Review the data product configuration:
4. Verify that all required assets are included
5. Confirm that the data product name and description are accurate
6. Click the Publish button on the details page
7. The system will publish the data product, making it active and effective

Before Publishing:

Ensure your data product is ready for publication:

• Assets Added: Verify that all required assets have been added to the data product
• Configuration Complete: Confirm that the data product name and description accurately represent its purpose
• Review: Review the data product configuration to ensure it meets your requirements

After Publishing:

• The data product status will change to Published
• The data product becomes available for access through the configured policies
• Users can access the logical data product view across different platforms
• Policies are dynamically applied based on user context and data attributes
• The default tag and Ranger role are available for use in policy definitions

Deleting a Data Product¶

To delete a data product:

1. Navigate to Governance Hub → Products
2. Click on the data product name/tab to open its details page
3. On the details page, click the Delete button
4. In the confirmation dialog:
5. Review the warning message about the deletion
6. Confirm that you want to delete the data product
7. Click Delete or Confirm to proceed with deletion

Managing Data Products¶

The Data Products interface provides comprehensive management capabilities for viewing, editing, and monitoring your data products.

Data Product Details View¶

When you click on a data product name/tab, the details page provides comprehensive information organized into two tabs:

Basic Information Tab¶

The Basic Information tab displays general information about the data product:

• Name: Business-aligned name of the data product
• Description: Detailed description of what the data product represents
• Tags: Tags assigned to the data product for classification and organization
• Created By: User who created the data product
• Create Time: Timestamp when the data product was created
• Updated By: User who last updated the data product
• Update Time: Timestamp of the most recent update

Assets Tab¶

The Assets tab shows all assets associated with the data product and provides asset management capabilities:

Viewing Assets:

• Asset List: Complete list of all assets (tables, columns, and other resources) associated with the data product
• Asset Information: For each asset, you can see the following details:
  • Asset name and path
  • Source data source or service
  • Asset type (table, column, view, etc.)

Search and Filter:

• Search by Data Sources or Services: Filter assets by selecting specific data sources or services
• Search by Asset Name: Enter keywords to search for assets by name
• Use these search and filter options to quickly find specific assets within the data product

Adding Assets:

1. Click the Add Assets button on the Assets tab
2. A dialog opens with the Data Catalog Explore view showing all available services and assets in Omni
3. Navigate through the hierarchy to browse and select the assets you want to add
4. Select one or more assets from the Data Catalog Explore view
5. Click Add Assets to add the selected assets to the data product

Removing Assets:

1. In the Assets tab, select one or more assets from the asset list
2. Click the Remove Selected button to remove all selected assets
3. The selected assets will be removed from the data product

Viewing Data Product Details¶

To view detailed information about a data product:

1. Navigate to Governance Hub → Products
2. Click on a data product name/tab to open its details page
3. Review the information in the tabs:
4. Basic Information: View name, description, tags, and metadata
5. Assets: View and manage all assets associated with the data product
6. Use the tabs to navigate between different aspects of the data product

Editing Data Product Information¶

To edit a data product's name, description, or other metadata:

1. Navigate to Governance Hub → Products
2. Click on the data product name/tab to open its details page
3. Click Edit or the edit icon on the details page
4. Modify the desired fields:
5. Name: Update the data product name (if allowed)
6. Description: Modify the description
7. Click Save to apply the changes

Data Product Workflow¶

A typical data product lifecycle follows these steps:

Creating and Publishing a Data Product¶

1. Create: Define a new data product with a business-aligned name and description
2. Add Assets: Associate physical data assets (tables, columns) with the data product
3. Configure: Set up tags, metadata, and ensure policies are configured
4. Review: Verify the data product configuration meets requirements
5. Publish: Make the data product active and available for use
6. Monitor: Track usage and manage the data product over time

Managing Published Data Products¶

• Monitor Usage: Track how the data product is being accessed
• Update Assets: Add or remove assets as business needs change
• Modify Policies: Update access policies to reflect changing requirements
• Maintain: Keep descriptions and metadata current

Best Practices¶

Naming Conventions¶

Follow consistent naming conventions to ensure clarity and discoverability:

• Business-Aligned Names: Use clear, business-aligned names that reflect the logical purpose of the data product
• Good: "Customer Data", "Health Claims", "Transaction Logs"
• Avoid: "California_Customers", "Restricted_Health_Data" (encode constraints in names)
• Consistent Patterns: Use consistent naming patterns across your organization
• Descriptive: Names should clearly indicate what the data product represents
• Generic: Keep names generic to allow flexibility in policy application

Asset Organization¶

Organize assets effectively within data products:

• Logical Grouping: Group related assets logically within a data product
• Business Use Case: Consider the business use case when selecting assets
• Cross-Platform: Ensure assets from different platforms can be meaningfully combined
• Completeness: Include all assets needed to represent the complete business view
• Maintainability: Organize assets in a way that makes the data product easy to maintain

Policy Configuration¶

Leverage Privacera's policy engine effectively:

• Policy-Driven: Use Privacera's policy engine for access control rather than hardcoding constraints
• Tag-Based Policies: Use tag-based policies for flexible, scalable access control
• Context-Aware: Configure policies that adapt to different usage contexts
• Dynamic Application: Let policies dynamically apply restrictions based on attributes and context
• Unified Governance: Manage policies alongside data product definitions for consistent governance

Use Cases and Examples¶

Data Products enable various use cases that benefit from logical abstraction and policy-driven access:

Cross-Platform Data Access¶

Scenario: Access customer data stored in a data lake through different query engines.

• Create a "Customer Data" data product
• Add customer tables from the data lake
• Access the same logical view through Snowflake, Databricks, or other platforms
• Policies automatically apply regardless of the access method

Unified Business Views¶

Scenario: Combine related data from multiple sources into a single business view.

• Create a "Sales Analytics" data product
• Add sales tables from different regions or systems
• Provide a unified view for analytics teams
• Policies ensure appropriate access based on user context

Policy-Driven Access Control¶

Scenario: Apply different access restrictions based on user context without creating multiple data products.

• Create a generic "Health Claims" data product
• Configure policies that apply restrictions based on:
• User role (e.g., analysts vs. administrators)
• Geographic location (e.g., regional restrictions)
• Compliance requirements (e.g., HIPAA, GDPR)
• Single data product, multiple access scenarios

Next Steps¶

• Learn about Data Catalog features
• Understand Tags and Metadata management
• Explore Access Management policies
• Learn about Tag-Based Access Control

Related Topics¶

• Governance Hub Overview
• Configuring Omni with PolicySync Connectors