Resource Loading Failure - GCP BigQuery¶

This guide covers resource loading failures when working with GCP BigQuery connector, including authentication, authorization, and API quota limit issues.

How to Check Logs for Issue Identification¶

Step 1: Log in to the jumpbox where Privacera Manager (PM) is installed

Step 2: List all running pods in the specified namespace

1

kubectl get pods -n <NAMESPACE>

Step 3: Open an interactive shell session inside the desired connector pod

1

kubectl exec -it <POD_NAME> -n <NAMESPACE> -- /bin/bash

Step 4: Navigate to PolicySync Logs Directory

1

cd /workdir/policysync/logs

Step 5: Check for API Quota Limit Issues

1
2

# Search for quota-related errors
grep -rnw "RESOURCE_EXHAUSTED\|Quota exceeded\|Rate limit" ./policy-sync.log

Step 6: Check for Authentication & Authorization Issues

1
2

# Search for authentication/authorization errors
grep -rnw "Access Denied\|Forbidden\|Unauthorized\|Permission denied\|Invalid authentication" ./policy-sync.log

Step 7: Analyze Results and Choose Troubleshooting Path

Based on your log search results:

• If you found RESOURCE_EXHAUSTED or quota-related errors → Go to API Limits & Quotas section
• If you found Access Denied, Forbidden, or authentication errors → Go to Authentication & Authorization Issues section

Authentication & Authorization Issues¶

Root Cause¶

• Service account key expired or missing permissions
• Missing or incorrectly configured PrivaceraPolicySyncRole IAM role
• Insufficient BigQuery/Resource Manager permissions
• Cross-project access not properly configured
• Workload Identity misconfiguration (for GCP instances)
• Service account not attached to correct projects/organization
• Missing required permissions for BigQuery Data Policies, Row Access Policies, or Data Catalog operations

Resolutions¶

1. Verify service account key validity and regenerate if needed

   • Check service account key expiration date:

     • Navigate to GCP Console → IAM & Admin → Service Accounts
     • Click on the service account used by your BigQuery connector
     • Go to the "Keys" tab to view all keys and their creation dates
     • Check if any keys are approaching expiration (keys don't have explicit expiration dates but may be compromised or rotated for security)
     • Look for keys that are very old (> 90 days) or known to be compromised

   • Generate a new service account key:

     • In the same "Keys" tab, click "ADD KEY" → "Create new key"
     • Select "JSON" format and click "CREATE"
     • The new key file will be automatically downloaded to your local machine
     • Important: Store this JSON file securely as it contains sensitive credentials

   • Update the configuration with the new service account key:

     Self Managed (Portal Configuration)Self Managed (YAML Configuration) and Data Plane

     • The service account key is managed automatically when you specify the "Service Account Email"
     • If using a custom key, you'll need to update the infrastructure-related properties and download the updated ZIP file
     • Refer to the Setup Guide for detailed steps on updating portal configurations

     • Copy the new JSON key file to the connector directory:

       Bash
       1	cp /path/to/new-service-account-key.json ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/

     • Update the vars.connector.bigquery.yml file:

       YAML
       1	CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME: "new-service-account-key.json"

     • Run the update command:

       Bash
       1 2 3	cd ~/privacera/privacera-manager ./privacera-manager.sh setup ./pm_with_helm.sh upgrade

   • Delete the old service account key: - Once the new key is working, delete the old key from GCP Console for security - Remove the old JSON file from your local system and the Privacera Manager instance

2. Verify PrivaceraPolicySyncRole IAM role exists and is properly configured

   • Ensure the PrivaceraPolicySyncRole is created at project or organization level
   • Verify the role includes all required permissions as specified in Prerequisites
   • Check that the role includes essential permissions like:
     • resourcemanager.projects.get
     • bigquery.datasets.getIamPolicy, bigquery.datasets.setIamPolicy
     • bigquery.tables.getIamPolicy, bigquery.tables.setIamPolicy
     • bigquery.dataPolicies.* permissions
     • bigquery.rowAccessPolicies.* permissions

3. Verify service account has proper role assignments

   • Navigate to GCP Console → IAM & Admin → IAM
   • Ensure the service account has the PrivaceraPolicySyncRole assigned
   • For multi-project setups, verify the service account has access to all required projects
   • Check both project-level and organization-level role assignments if applicable

4. Configure cross-project permissions properly

   • Option 1: Use organization-level role for managing multiple projects
     • Create PrivaceraPolicySyncRole at organization level
     • Assign the role to service account at organization level
   • Option 2: Use project-level roles for each individual project
     • Create PrivaceraPolicySyncRole in each target project
     • Assign the role to service account in each project
   • Refer to Prerequisites for detailed setup

5. Verify Workload Identity configuration (for GCP instances)

   • Check if Workload Identity is properly enabled and configured
   • Ensure the GCP instance has the correct service account attached
   • Verify Workload Identity binding between Kubernetes service account and GCP service account
   • See Workload Identity setup for configuration details

6. Check basic authentication setup (for non-GCP instances)

   • Verify the service account JSON key file is correctly placed and accessible
   • Ensure the JSON file contains all required fields and valid credentials
   • Confirm the file path is correctly specified in the connector configuration
   • Refer to Basic Authentication setup

API Limits & Quotas¶

Root Cause¶

• Service API rate limits exceeded
• BigQuery Data Policy API - Read requests per minute quota exceeded
• BigQuery Data Policy API - Write requests per minute quota exceeded

Resolutions¶

1. Check quota usage in GCP Console → IAM & Admin → Quotas

   • Identify which quota limits are being exceeded
   • Monitor usage patterns and trends

2. Request quota increase in GCP Console

   • Submit quota increase request for the specific API
   • Note: Quota increases are subject to Google's approval and may be denied

3. If quota increase is denied or doesn't resolve the issue, escalate to support

   • See the Escalation Guidelines for detailed information

Comments