Prerequisites for BigQuery Connector - Access Management¶
GCP BigQuery connector with Privacera requires the following prerequisites:
Mandatory Prerequisites¶
Prerequisites | Detail |
---|---|
IAM Role Permissions for managing BigQuery Refer | This is used to create Permissions for GCP IAM Role. |
Create PrivaceraPolicySyncRole IAM Role for Project Refer | This is used create GCP IAM role at project-level for service account. |
Create PrivaceraPolicySyncRole IAM Role for organization Refer | This is used to create GCP IAM role at organization-level for service account. |
Attach IAM Role to Service Account Refer | This is used to connect policysync with GCP bigquery. |
Configure Logs for Auditing Refer | This is used to store audits logs of GCP BigQuery to load by policysync. |
Optional Prerequisites¶
You must choose one of the following options for setting up the BigQuery connector:
Prerequisites | Detail |
---|---|
Enable WorkLoad Identity for BigQuery Refer (optional) | If you are deploying the connector on a GCP instance, you can use Workload Identity to authenticate. |
Basic Authentication for PolicySync Refer (optional) | If you are using other instances (e.g., EKS, AKS), choose this option of basic authentication. |
Managing Multiple GCP Projects with a Single BigQuery Connector¶
If you have multiple projects within your GCP organization and wish to manage them using a single BigQuery connector, you can follow one of the two options below. Choose the option that best fits your needs.
Option 1: Service Account at One Project with Organization-level Role¶
-
Create Organization-level IAM Role:
- Create an IAM role at the organization level that will allow the service account to access BigQuery across all projects.
- IAM Role Permissions for managing BigQuery
- Create Organization-level IAM Role for Service Account
-
Assign the Role to the Service Account:
- Create a service account in any one of the projects within your organization.
- Assign the organization-level IAM role to the service account, granting it the necessary permissions to manage BigQuery across the organization.
- This ensures the service account can access BigQuery resources in all projects within the organization.
- Attach IAM Role to Service Account
Option 2: Service Account at One Project with Project-level Role for Each Project¶
-
Create IAM Role at Every Project:
- Create an IAM role at the project level for each project where the service account needs to access BigQuery. This allows the service account to be granted permissions to specific resources in each project.
- IAM Role Permissions for managing BigQuery
- Create Project-level Access IAM Role for Service Account
-
Assign the Role to the Service Account:
- Create a service account in any one of the projects within your organization.
- Assign the project-level IAM role to the service account for each individual project. This ensures the service account has the required permissions to access BigQuery in all the relevant projects.
- Attach IAM Role to Service Account
Appendix¶
IAM Role Permissions for managing BigQuery¶
You need to give Privacera PolicySync basic access to GCP. To grant that access, create PrivaceraPolicySyncRole IAM role in your GCP project or GCP organization using the following commands on Google Cloud's shell (gcloud). The shell can be installed and accessed locally or through Google Console.
Run the following command to create the file containing the permissions required for the PrivaceraPolicySyncRole role:
Create Project-level Access IAM Role for Service Account¶
To create the PrivaceraPolicySyncRole in your GCP project, execute the following command. Replace <GCP_PROJECT_ID>
with your actual GCP project ID:
PROJECT_ID="<GCP_PROJECT_ID>"
To create PrivaceraPolicySyncRole role in your GCP project, run the following command.
Text Only | |
---|---|
Create Organization-level IAM Role for Service Account¶
To create the PrivaceraPolicySyncRole role in your GCP organization, run the following command. Replace <GCP_ORGANIZATION_ID>
with your GCP organization ID:
ORGANIZATION_ID="<GCP_ORGANIZATION_ID>"
To create PrivaceraPolicySyncRole role in your GCP organization, run the following command.
Text Only | |
---|---|
Attach IAM Role to Service Account¶
To attach the PrivaceraPolicySyncRole IAM role created above, please follow the below steps:
-
Log in to your GCP console.
-
Navigate to IAM & admin > Service accounts, and click + CREATE SERVICE ACCOUNT.
-
Enter the required values in the fields and click CREATE.
-
In the Grant this service account access to project section, select the role
PrivaceraPolicySyncRole
. -
On the Service Account Page, locate the newly created service account and copy its email address for use in a later step.
-
If you are using a Google VM instance to configure Google BigQuery for PolicySync, you may attach the service account created above to your VM instance and skip the following steps.
-
On the Service Account Page, navigate to the Keys tab, click Add Key, and then select Create New Key.
-
Select the JSON key type and click CREATE. A JSON key file will be downloaded to your system. Please store the file in an accessible location, as it will be used to configure PolicySync in Privacera Manager.
For more detailed information on creating a service account, see the Google documentation.
Configure Logs for Auditing¶
A sink is required to collect all logs from Google BigQuery. To create a sink, please follow the below steps:
-
In the search bar, type Logging, then click on Logs Router and select Create Sink.
-
Enter the sink name as
PolicySyncBigQueryAuditSink
, and then click Next. -
Enter the sink destination.
-
In the Select sink service section, choose BigQuery.
-
In the Select BigQuery dataset section, click Create new BigQuery dataset.
-
Enter the Dataset ID as
bigquery_audits
and click Create Dataset. -
Click Next.
-
Add the BigQuery logs in the sink:
- In the Build an inclusion filter, add the following line:
Text Only
- In the Build an inclusion filter, add the following line:
-
Click Create Sink.
For more detailed information on creating a sink, see the Google documentation.
Workload Identity for BigQuery (Optional)¶
To enable Workload Identity for BigQuery, please follow the steps below:
-
Enable Workload Identity on the GKE cluster:
-
To enable Workload Identity Federation for GKE on an existing cluster, run the following command:
Replace the following:Text Only - CLUSTER_NAME: the name of your new cluster.
- LOCATION: the Compute Engine location for the cluster.
- PROJECT_ID: your Google Cloud project ID.
-
To modify an existing node pool to use Workload Identity Federation for GKE, run the following command:
Replace the following:Text Only - NODEPOOL_NAME: the name of the new node pool.
- CLUSTER_NAME: the name of the existing cluster with Workload Identity Federation for GKE enabled.
- REGION: the region where the cluster is located.
-
-
Create an IAM policy that allows the Kubernetes ServiceAccount to impersonate the IAM service account::
- Run the following command to add the IAM policy binding: Replace the following:
Text Only - PROJECT_ID: your Google Cloud project ID.
- NAMESPACE: the Kubernetes namespace name.
- KSA_NAME: the Kubernetes ServiceAccount name Eg.
connector-<CONNECTOR-NAME>-<INSTANCE-NAME>-privacera-sa
. - IAM_SA_NAME: the IAM service account name.
- Run the following command to add the IAM policy binding:
Note
If you don't have an existing cluster in your GCP project, please follow the instructions at this link Refer to create a new cluster and node pool.
-
Enable Workload Identity on the GKE cluster:
-
To enable Workload Identity Federation for GKE on an existing cluster:
- Navigate to Kubernetes Engine > Clusters.
- Select the cluster name.
- In the Workload Identity section, Click Edit, select the Enable Workload Identity checkbox.
- Click Save.
-
To modify an existing node pool to use Workload Identity Federation for GKE, perform the following steps:
- Navigate to Kubernetes Engine > Clusters.
- Click on the cluster name.
- Click on the Node tab.
- Click on the node pool name you want to modify.
- Click Edit.
- In the Security section, select the Enable GKE Metadata Server checkbox.
- Click Save.
-
-
Create an IAM allow policy that grants the Kubernetes ServiceAccount permission to impersonate the IAM service account:
- Go to the IAM & Admin page in the Google Cloud Console.
- Click on Service accounts in the sidebar.
- Find and select your service account.
- Go to the Permissions tab.
- Click Grant Access, and in the role section, search for workload identity user.
- In Principal name add
<PROJECT-ID>.svc.id.goog[<NAMESPACE-NAME>/<KSA_NAME>]
.- PROJECT_ID: your Google Cloud project ID.
- Namespace-name: the Kubernetes namespace name.
- KSA_NAME: the Kubernetes ServiceAccount name Eg.
connector-<connector-name>-<Instance-Name>-privacera-sa
.
- Click Save.
Note
If you don’t have an existing cluster in your GCP project, follow the instructions at this link Refer to create a new cluster and node pool.
Basic Authentication for PolicySync (Optional)¶
To optionally enable basic authentication for PolicySync with Google BigQuery, create a JSON file in the subdirectory of your connector instance. The file must be named XXX.json.
An example of the contents of XXX.json.
- Prev topic: Push Mode Overview
- Next topic: Setup