Advanced Configuration for BigQuery Connector - Access Management¶

Custom IAM Roles¶

Goal¶

Facilitate fine-grained access control within your BigQuery integration by utilizing custom IAM roles that are automatically created by PolicySync, or alternatively, manage them manually.

Prerequisites¶

1. You have successfully installed Privacera Manager and have the base installation operational.
2. You have configured the connector for BigQuery or are in the process of doing so.

Steps¶

Please modify the following properties in the vars.connector.bigquery.yml file located in the instance directory of the connector.

1
2
3
4
5
6
7
8

# Enable automatic creation of custom IAM roles in your GCP project or organization
CONNECTOR_BIGQUERY_CREATE_CUSTOM_IAM_ROLES: "true"

# Define whether custom IAM roles should be created at the project or organization level
CONNECTOR_BIGQUERY_CUSTOM_IAM_ROLES_SCOPE: "project"  # or "org"

# Set your GCP organization ID if using organization-level custom IAM roles
CONNECTOR_BIGQUERY_ORGANIZATION_ID: "your-gcp-org-id"

1. CONNECTOR_BIGQUERY_CREATE_CUSTOM_IAM_ROLES

   • Enable this property if you wish for PolicySync to automatically create custom IAM roles in your GCP project or organization. This will facilitate fine-grained access control.

   • Important

     If this property is disabled (false), you will need to create all custom IAM roles manually within your GCP project or organization.

2. CONNECTOR_BIGQUERY_CUSTOM_IAM_ROLES_SCOPE

   • This property defines the scope in which custom IAM roles will be created and utilized.
   • Set this property to project if you wish for custom IAM roles to be created and utilized at the individual project level.
   • Set this property to org if you wish for custom IAM roles to be created and utilized at the organization level.
   • If you wish to create custom IAM roles manually at the organization level, they will be used across all managed projects within that organization.

3. CONNECTOR_BIGQUERY_ORGANIZATION_ID

   • If you opt to use organization-level IAM roles (by setting CONNECTOR_BIGQUERY_CUSTOM_IAM_ROLES_SCOPE to org), please specify your GCP organization ID in this field.

   • Note

     This property is required only when custom IAM roles are created at the organization level.

Filter database, schema, and tables from access management¶

Goal¶

1. You wish to manage access permissions for a specific dataset and table in BigQuery.
2. You wish to exclude specific datasets and tables from access management in BigQuery.
3. You wish to manage access permissions for all tables within BigQuery.

Prerequisites¶

1. You have successfully installed Privacera Manager and have the base installation operational.
2. You have configured the connector for BigQuery or are in the process of doing so.

Steps¶

Please modify the following properties in the vars.connector.bigquery.yml file located in the instance directory of the connector:

1
2
3
4
5
6
7

CONNECTOR_BIGQUERY_MANAGE_PROJECT_LIST: "gcp-project-123"
CONNECTOR_BIGQUERY_MANAGE_DATASET_LIST: "gcp-project-123.analytics_db"
CONNECTOR_BIGQUERY_MANAGE_TABLE_LIST: "gcp-project-123.analytics_db.customer_table, gcp-project-123.analytics_db.finance_*"

CONNECTOR_BIGQUERY_IGNORE_PROJECT_LIST: "gcp-project-111"
CONNECTOR_REDSHIFT_IGNORE_DATABASE_LIST: "gcp-project-123.test_db"
CONNECTOR_REDSHIFT_IGNORE_TABLE_LIST: "*.*.test_*"

1. You can provide a comma-separated list of project, dataset and table names.
2. You can use * as a wildcard character.
3. For dataset, the format is <project_id>.<dataset>.
4. For table, the format is <project_id>.<dataset>.<table>.
5. The ignore list takes precedence over the manage list. If a project, dataset, or table appears in both lists, it will be excluded.
6. These entries are case-sensitive.
7. If you set a MANAGE list value to empty, it will manage all objects of that type. You can use this to manage all objects within a project.

Filtering Privacera user, group, and role names¶

Goal¶

You wish to manage access permissions for a specific set of Privacera users, groups, and roles within your BigQuery environment.

Prerequisites¶

1. You have successfully installed Privacera Manager and have the base installation operational.
2. You have configured the connector for BigQuery or are in the process of doing so.

Steps¶

Please modify the following properties in the vars.connector.bigquery.yml file located in the instance directory of the connector:

1
2
3
4
5
6
7

CONNECTOR_BIGQUERY_MANAGE_USER_LIST: "privacera_user1, privacera_user2"
CONNECTOR_BIGQUERY_MANAGE_GROUP_LIST: "privacera_group1, privacera_test_group_*"
CONNECTOR_BIGQUERY_MANAGE_ROLE_LIST: "privacera_role1, privacera_test_role_*"

CONNECTOR_BIGQUERY_IGNORE_USER_LIST: "test_user1"
CONNECTOR_BIGQUERY_IGNORE_GROUP_LIST: "test_group1"
CONNECTOR_BIGQUERY_IGNORE_ROLE_LIST: "test_role1"

1. You can provide a comma-separated list of user, group, and role names.
2. You can use * as a wildcard character.
3. The ignore list takes precedence over the manage list. If a user, group, or role appears in both lists, it will be excluded.
4. If you wish to manage all users, you can skip specifying these properties.

Access Control Management¶

Goal¶

Enable advanced access control management for BigQuery by configuring row filters and masking policies using native BigQuery functionality or secure views.

Prerequisites¶

1. You have successfully installed Privacera Manager and have the base installation operational.
2. You have configured the connector for BigQuery or are in the process of doing so.

Native Row Filter and Tag Masking¶

Steps¶

Modify the following properties in the vars.connector.bigquery.yml file located in the instance directory of the connector.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Enable native BigQuery tag masking
ENABLE_TAG_MASKING: "true"

# Enable native BigQuery row filters
CONNECTOR_BIGQUERY_ENABLE_ROW_FILTER: "true"

# Enable view-based masking for BigQuery
CONNECTOR_BIGQUERY_ENABLE_VIEW_BASED_MASKING: "false"

# Create secure views for all tables and views, regardless of policies
CONNECTOR_BIGQUERY_SECURE_VIEW_CREATE_FOR_ALL: "false"

# Enable view-based row filters for BigQuery
CONNECTOR_BIGQUERY_ENABLE_VIEW_BASED_ROW_FILTER`: "false"

CONNECTOR_BIGQUERY_COLUMN_ACCESS_CONTROL_TYPE`: "none"

CONNECTOR_BIGQUERY_ENABLE_DATA_ADMIN: "false"

Secure View Row Filter and Masking¶

Steps¶

Modify the following properties in the vars.connector.bigquery.yml file located in the instance directory of the connector.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Enable native BigQuery row filters
CONNECTOR_BIGQUERY_ENABLE_ROW_FILTER: "false"

# Enable native BigQuery tag masking
ENABLE_TAG_MASKING: "false"

# Enable view-based masking for BigQuery
CONNECTOR_BIGQUERY_ENABLE_VIEW_BASED_MASKING: "true"

# Enable view-based row filters for BigQuery
CONNECTOR_BIGQUERY_ENABLE_VIEW_BASED_ROW_FILTER: "true"

# Create secure views for all tables and views, regardless of policies
CONNECTOR_BIGQUERY_SECURE_VIEW_CREATE_FOR_ALL: "false"

The following properties define how access control mechanisms are managed within BigQuery:

1. CONNECTOR_BIGQUERY_ENABLE_ROW_FILTER

   • Description: Enables native BigQuery row filters.
   • Recommended Setting: false

2. CONNECTOR_BIGQUERY_ENABLE_VIEW_BASED_MASKING

   • Description: Enables masking policies using secure views.
   • Recommended Setting: true

   • Recommendation

     Text Only
     1	View-based masking is preferred since BigQuery does not support native masking.

3. CONNECTOR_BIGQUERY_ENABLE_VIEW_BASED_ROW_FILTER

   • Description: Enables row filter policies using secure views.
   • Recommended Setting: true

4. CONNECTOR_BIGQUERY_SECURE_VIEW_CREATE_FOR_ALL

   • Description: Creates secure views for all tables and views, regardless of existing masking or row filter policies.
   • Recommended Setting: false

Access Audits Management¶

Goal¶

Enable and manage access audits in BigQuery through PolicySync, allowing detailed tracking and filtering of access events.

Prerequisites¶

1. You have successfully installed Privacera Manager and have the base installation operational.
2. You have configured the connector for BigQuery or are in the process of doing so.

Steps¶

Please modify the following properties in the vars.connector.bigquery.yml file located in the instance directory of the connector.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Enable access audit fetching from BigQuery
CONNECTOR_BIGQUERY_AUDIT_ENABLE: "true"

# Set the list of users whose access audits should be ignored
CONNECTOR_BIGQUERY_AUDIT_EXCLUDED_USERS: "user1@example.com, user2@example.com"

# Set the project ID to fetch BigQuery audits
CONNECTOR_BIGQUERY_AUDIT_PROJECT_ID: "your-bigquery-project-id"

# Set the dataset name to fetch BigQuery audits
CONNECTOR_BIGQUERY_AUDIT_DATASET_NAME: "your-bigquery-dataset-name"

1. CONNECTOR_BIGQUERY_AUDIT_ENABLE

   • Set this property to true if you want to enable access audit fetching from BigQuery.
   • When enabled, PolicySync will gather access audit data from the specified BigQuery project and dataset.

2. CONNECTOR_BIGQUERY_AUDIT_EXCLUDED_USERS

   • This property is used to specify a list of users whose access audits should be excluded by PolicySync.
   • Provide a comma-separated list of email addresses for users whose access events should be excluded from the audit logs.

3. CONNECTOR_BIGQUERY_AUDIT_PROJECT_ID

   • Specify the project ID from which BigQuery audits should be retrieved.
   • This is the GCP project where the audit logs are stored and queried.

4. CONNECTOR_BIGQUERY_AUDIT_DATASET_NAME

   • Specify the dataset name that will be used to query and retrieve access audits from BigQuery.
   • This is the dataset within the project that contains the audit information.

1. In PrivaceraCloud, go to Settings -> Applications.

2. On the Applications screen, select BigQuery.

3. Enter the application Name and Description. Click Save. Name could be any name of your choice. E.g. BigQuery Connector for account 123456.

4. Open the BigQuery application.

5. Enable the Access Management option with toggle button.

6. Under the ADVANCED tab, enter the values for:

   • Create Custom IAM Roles in GCP : Specifies whether PolicySync automatically creates custom IAM roles in your Google Cloud Platform project or organization for fine-grained access control (FGAC). If set to false, you must create all required custom IAM roles manually in your GCP project or organization. Default value is true.

   • GCP Custom IAM Roles Scope : Specifies whether PolicySync creates and uses custom IAM roles at the project or organizational level. Allowed values: project (for individual project-level roles) or org (for organization-level roles).

   • GCP Organization ID : Specifies the GCP organization ID. Provide this value if you are configuring PolicySync to use custom IAM roles at the organizational level.

   • Datasets to Set Access Control Policies : Specifies a list of comma-separated datasets that PolicySync manages access control to. Wildcards can be used. If no value is set, all datasets are managed.

   • Tables to Set Access Control Policies : Specifies a comma-separated list of table names for which PolicySync manages access control. Use the format <PROJECT_NAME>.<DATASET_NAME>.<TABLE_NAME>. Wildcards can be used.

   • Projects to Ignore While Setting Access Control Policies : Specifies a comma-separated list of project names that PolicySync does not provide access control for. Wildcards are allowed.

   • Datasets to Ignore While Setting Access Control Policies : Specifies a comma-separated list of dataset names that PolicySync does not provide access control for. Wildcards are allowed.

   • Tables to Ignore While Setting Access Control Policies : Specifies a comma-separated list of table names that PolicySync does not provide access control for. Wildcards are allowed.

   • Users to Set Access Control Policies : Specifies a comma-separated list of user names for which PolicySync manages access control. Wildcards can be used.

   • Groups to Set Access Control Policies : Specifies a comma-separated list of group names for which PolicySync manages access control. Wildcards can be used.

   • Users to be Ignored by Access Control Policies : Specifies a comma-separated list of user names that PolicySync does not provide access control for. Wildcards can be used.

   • Groups to be Ignored by Access Control Policies : Specifies a comma-separated list of group names that PolicySync does not provide access control for. Wildcards can be used.

   • Set Access Control Policies Only on Users from Managed Groups : Specifies whether to manage only the users that are members of groups specified by Groups to set access control policies. Default value is false.

   • Enforce BigQuery Native Row Filter : Specifies whether to use the data source native row filter functionality. Disabled by default.

   • Enforce Masking Policies Using Secure Views : Specifies whether to use secure view-based masking. Default value is true.

   • Enforce Row Filter Policies Using Secure Views : Specifies whether to use secure view-based row filtering. Default value is true.

   • Create Secure View for All Tables/Views : Specifies whether to create secure views for all tables and views. Enabled by default.

   • Default Masking Value for Numeric Datatype : Specifies the masking value used for numeric data types.

   • Default Masking Value for Text/String Datatype : Specifies the masking value used for text or string data types.

   • Secure View Name Prefix : Specifies a prefix string for secure views.

   • Secure View Name Postfix : Specifies a postfix string for secure views.

   • Secure View Dataset Name Prefix : Specifies a prefix string for the secure view dataset name.

   • Secure View Dataset Name Postfix : Specifies a postfix string for the secure view dataset name.

   • Enable this for Policy Enforcements and User/Group/Role Management : Specifies whether PolicySync performs grants and revokes for access control and manages users, groups, and roles. Default value is true.

   • Enable to Use Data Admin Functionality : Enables data admin functionality for creating policies on native tables/views with row filter and masking capability.

   • Ignore Audit for Users : Specifies a comma-separated list of users to exclude from access audits.

   • Project ID Used to Fetch BigQuery Audits : Specifies the project ID where BigQuery audit logs are stored.

   • Dataset Used to Fetch BigQuery Audits : Specifies the name of the dataset where BigQuery logs audit data.

7. Click SAVE.

8. The configured BigQuery connector appears under Applications.

9. Once saved and enabled, the BigQuery connector will start. Then you can hover on the VIEW LOGS button to check the status, either Running or Stopped.

10. Perform following steps to restart the BigQuery connector application:

    1. Go to Settings → Applications → select the****BigQuery** connector application** .

    2. Edit the application → Disable it → and Save it.

    3. Open the same application again and then: Enable it → and Save it.