Prerequisites for BigQuery Connector - Access Management¶

GCP BigQuery connector with Privacera requires the following prerequisites:

Mandatory Prerequisites¶

Optional Prerequisites¶

You must choose one of the following options for setting up the BigQuery connector:

Managing Multiple GCP Projects with a Single BigQuery Connector¶

If you have multiple projects within your GCP organization and wish to manage them using a single BigQuery connector, you can follow one of the two options below. Choose the option that best fits your needs.

Option 1: Service Account at One Project with Organization-level Role¶

1. Create Organization-level IAM Role:

   • Create an IAM role at the organization level that will allow the service account to access BigQuery across all projects.
   • IAM Role Permissions for managing BigQuery
   • Create Organization-level IAM Role for Service Account

2. Assign the Role to the Service Account:

   • Create a service account in any one of the projects within your organization.
   • Assign the organization-level IAM role to the service account, granting it the necessary permissions to manage BigQuery across the organization.
   • This ensures the service account can access BigQuery resources in all projects within the organization.
   • Attach IAM Role to Service Account

Option 2: Service Account at One Project with Project-level Role for Each Project¶

1. Create IAM Role at Every Project:

   • Create an IAM role at the project level for each project where the service account needs to access BigQuery. This allows the service account to be granted permissions to specific resources in each project.
   • IAM Role Permissions for managing BigQuery
   • Create Project-level Access IAM Role for Service Account

2. Assign the Role to the Service Account:

   • Create a service account in any one of the projects within your organization.
   • Assign the project-level IAM role to the service account for each individual project. This ensures the service account has the required permissions to access BigQuery in all the relevant projects.
   • Attach IAM Role to Service Account

Appendix¶

IAM Role Permissions for managing BigQuery¶

You need to give Privacera PolicySync basic access to GCP. To grant that access, create PrivaceraPolicySyncRole IAM role in your GCP project or GCP organization using the following commands on Google Cloud's shell (gcloud). The shell can be installed and accessed locally or through Google Console.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

ROLE_NAME="PrivaceraPolicySyncRole"

cat << EOF > ${ROLE_NAME}.yaml
title: "${ROLE_NAME}"
description: "${ROLE_NAME}"
stage: "ALPHA"
includedPermissions:
- resourcemanager.projects.get
- resourcemanager.projects.getIamPolicy
- resourcemanager.projects.setIamPolicy
- iam.roles.list
- iam.roles.get
- iam.roles.create
- iam.roles.update
- bigquery.jobs.create
- bigquery.datasets.get
- bigquery.datasets.create
- bigquery.datasets.update
- bigquery.datasets.delete
- bigquery.datasets.getIamPolicy
- bigquery.datasets.setIamPolicy
- bigquery.tables.list
- bigquery.tables.get
- bigquery.tables.getData
- bigquery.tables.create
- bigquery.tables.update
- bigquery.tables.delete
- bigquery.tables.getIamPolicy
- bigquery.tables.setIamPolicy
- bigquery.rowAccessPolicies.list
- bigquery.rowAccessPolicies.create
- bigquery.rowAccessPolicies.update
- bigquery.rowAccessPolicies.delete
- bigquery.rowAccessPolicies.getIamPolicy
- bigquery.rowAccessPolicies.setIamPolicy
- datacatalog.categories.getIamPolicy
- datacatalog.categories.setIamPolicy
- datacatalog.taxonomies.get
- datacatalog.taxonomies.getIamPolicy
- datacatalog.taxonomies.list
- datacatalog.taxonomies.setIamPolicy
- datacatalog.taxonomies.update
- bigquery.dataPolicies.create
- bigquery.dataPolicies.delete
- bigquery.dataPolicies.get
- bigquery.dataPolicies.getIamPolicy
- bigquery.dataPolicies.list
- bigquery.dataPolicies.setIamPolicy
- bigquery.dataPolicies.update
- bigquery.tables.setCategory

EOF

Here are the descriptions of the GBQ Policies

Create Project-level Access IAM Role for Service Account¶

To create the PrivaceraPolicySyncRole in your GCP project, execute the following command. Replace <GCP_PROJECT_ID> with your actual GCP project ID:
PROJECT_ID="<GCP_PROJECT_ID>"

1

gcloud iam roles create ${ROLE_NAME} --project=${PROJECT_ID} --file=${ROLE_NAME}.yaml    

Create Organization-level IAM Role for Service Account¶

To create the PrivaceraPolicySyncRole role in your GCP organization, run the following command. Replace <GCP_ORGANIZATION_ID> with your GCP organization ID:
ORGANIZATION_ID="<GCP_ORGANIZATION_ID>"

1

 gcloud iam roles create ${ROLE_NAME} --organization=${ORGANIZATION_ID} --file=${ROLE_NAME}.yaml

Attach IAM Role to Service Account¶

To attach the PrivaceraPolicySyncRole IAM role created above, please follow the below steps:

1. Log in to your GCP console.

2. Navigate to IAM & admin > Service accounts, and click + CREATE SERVICE ACCOUNT.

3. Enter the required values in the fields and click CREATE.

4. In the Grant this service account access to project section, select the role PrivaceraPolicySyncRole.

5. On the Service Account Page, locate the newly created service account and copy its email address for use in a later step.

6. If you are using a Google VM instance to configure Google BigQuery for PolicySync, you may attach the service account created above to your VM instance and skip the following steps.

7. On the Service Account Page, navigate to the Keys tab, click Add Key, and then select Create New Key.

8. Select the JSON key type and click CREATE. A JSON key file will be downloaded to your system. Please store the file in an accessible location, as it will be used to configure PolicySync in Privacera Manager.

For more detailed information on creating a service account, see the Google documentation.

Configure Logs for Auditing¶

A sink is required to collect all logs from Google BigQuery. To create a sink, follow the below steps:

1. In the search bar, type Logging, then click Log Router and select Create Sink.

2. Enter the sink name as PolicySyncBigQueryAuditSink, and click Next.

3. Configure the sink destination:

   1. Under Select sink service, choose BigQuery dataset.
   2. Under Select BigQuery dataset, click Create new BigQuery dataset.

   3. Enter the Dataset ID as bigquery_audits, and click Create Dataset.

      Warning

      Ensure the Use partitioned tables option remains unchecked while creating the sink.
      Enabling partitioned tables can cause issues with log retention and querying for auditing purposes.

   4. Click Next.

4. Add BigQuery logs to the sink:

   1. In the Build inclusion filter section, add the following line and click Next:

      Text Only
      1	resource.type="bigquery_resource"

5. Review the configuration and click Create Sink.

For more detailed information on creating a sink, see the Google documentation.

Workload Identity for BigQuery (Optional)¶

To enable Workload Identity for BigQuery, please follow the steps below:

Basic Authentication for PolicySync (Optional)¶

To optionally enable basic authentication for PolicySync with Google BigQuery, create a JSON file in the subdirectory of your connector instance. The file must be named XXX.json.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

 {
  "type": "service_account",
  "project_id": "your_project_id",
  "private_key_id": "autogenerated_value",
  "private_key": "-----BEGIN PRIVATE KEY-----autogenerated_value-----END PRIVATE KEY-----\n",
  "client_email": "autogenerated_value",
  "client_id": "autogenerated_value",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/autogenerated_value"
}

Enable Required Google Cloud APIs¶

Before configuring the BigQuery connector, ensure that the following Google Cloud APIs are enabled in your GCP project. These APIs are essential for the connector to function properly and perform access management operations.

Required APIs¶

How to Enable APIs¶

You can enable these APIs using either the Google Cloud Console or the gcloud CLI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Set your project ID
PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID

# Enable required APIs
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable iam.googleapis.com
gcloud services enable datacatalog.googleapis.com
gcloud services enable bigquerydatapolicy.googleapis.com
gcloud services enable bigqueryconnection.googleapis.com
gcloud services enable bigquery.googleapis.com