Skip to content

JDBC Error Rate Alerts

Root Cause

JDBC Error Rate alerts are triggered when more than 5% of JDBC calls fail within a 5-minute period. This alert indicates that your application is having trouble connecting to or interacting with the database. Common reasons include:

  • Database connection issues: Temporary network problems or the database service being unavailable.
  • Resource limits: The connection pool is exhausted, or the database server has run out of resources.
  • Query errors: Incorrect SQL statements, invalid table/column references, or schema mismatches.
  • Authentication failures: Wrong credentials or expired passwords.
  • Permission issues: The database user does not have the required privileges.
  • Database server problems: The server is overloaded, undergoing maintenance, or restarting.

Troubleshooting Steps

  • Use Diagnostics Tool: The Diagnostics Tool provides automated testing of connector functionality, including JDBC connectivity.

    • Open the Diagnostic Portal and navigate to Dashboard → Pods.
    • Select the connector pod from the available pods list.
    • Under the CURRENT TEST RESULTS tab, review the PyTest Report for the following checks:
      • test_databricks_server_reachable: verifies basic connectivity.
      • test_databricks_connection_using_jdbc: tests JDBC connection setup.
      • test_dummy_query_using_jdbc: validates query execution.
      • To check resource utilization, review:
        • test_diag_client_disk_space
        • test_diag_client_pod_cpu_utilization
        • test_jvm_process_cpu_utilization

  • Verify Authentication and Permissions:

    • In your connector configuration file, verify that the following authentication-related properties are set correctly.

    • Configuration File Location: privacera/privacera-manager/config/custom-vars/connectors/<connector-type>/<instance>/

      • JDBC URL: Property that defines the database connection endpoint (for example, CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_URL and CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_URL).
      • JDBC Username: Property that specifies the database username (for example, CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_USERNAME and CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_USERNAME).
      • JDBC Password / Access Token: Property that provides authentication credentials (for example, CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_PASSWORD and CONNECTOR_DATABRICKS_UNITY_CATALOG_API_ACCESS_TOKEN).

      Note

      For the exact property names and examples, refer to your connector-specific documentation.

    • Verify Database Permissions: Connect to your database using the same credentials and confirm that the user has the required privileges.

  • Investigate Query Issues: If authentication is correct but queries are failing, then look for common error patterns:

    • Syntax Errors: Download logs from the Diagnostic Portal and search locally for query syntax in the logs, refer Download Log from Diagnostic Portal section.
    • Missing Objects: Verify the existence of tables or views through the Service Explorer in the portal.
      • In the portal, navigate to Access Management → Service Explorer, select the configured connector, and browse the resources to confirm whether the expected objects are present.
    • Permission Denied: Check access rights by verifying resource policies on portal.

  • Check Resource Usage and Configuration:

    • Ensure timeout values are appropriate for your network latency and database response times.
    • Consider increasing timeouts if you are experiencing frequent connection failures by reviewing the connection pool and timeout settings in the connector configuration file.
    • Configuration File Location: privacera/privacera-manager/config/custom-vars/connectors/<connector-type>/<instance>/

    • Common JDBC Properties to Check:

      • Connection timeout: CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_CONNECTION_TIMEOUT_IN_MS
      • Connection pool size: CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_MAX_POOL_SIZE, CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_MAX_POOL_SIZE
      • Socket timeout: CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_SOCKET_TIMEOUT_IN_SECONDS, CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_SOCKET_TIMEOUT_IN_SECONDS

      Note

      For the exact property names and examples, refer to your connector-specific documentation.

    • Check connector pod resource utilization:

      • Review the Pod Monitoring dashboard under Dashboards → Infra-Dashboards in Grafana to check pod memory and CPU usage.
      • If memory issues are detected, adjust resources using Compute Sizing.

Escalation Checklist

If the issue cannot be resolved through the specific troubleshooting guides, escalate it to the Privacera support with the following details. For additional assistance, refer How to Contact Support for detailed guidance on reaching out to the support team.

  • Timestamp of the error: Include the exact time the alert was triggered
  • Grafana dashboard and alert screenshots:
    • Grafana → Dashboards → Application-Dashboards → connectors → common → Connector JDBC Metrics
    • Grafana → Alerting → Alert rules → JDBC Error Rate Alert

  • Connector Service Logs: Include any logs showing the JDBC errors or connection issues

    Option 1: Download Log from Diagnostic Portal (Recommended)

    1. Open the Diagnostic Portal and navigate to Dashboard → Pods.
    2. Select the connector pod from the available pods list.
    3. Click on the Logs tab and download logs by clicking on DOWNLOAD LOGS button.

    Option 2: Manual Log Collection (If Diagnostic service is not enabled)

    Bash
    1
2
3
4
5
6
7
8
    # Create log archive
kubectl exec -it <POD> -n <NAMESPACE> -- bash -c "cd /workdir/policysync/logs/ && tar -czf connector-logs.tar.gz *.log"

# Copy the fixed-name archive
kubectl cp <POD>:/workdir/policysync/logs/connector-logs.tar.gz ./connector-logs.tar.gz -n <NAMESPACE>

# Extract logs
tar -xzf connector-logs.tar.gz

Comments