Skip to content

PToken Generation Error Rate Alert

Root Cause

A PToken Generation Error Rate alert is triggered when more than 5% of PToken generation requests fail within a 5-minute period. PToken generation is required for secure access to services, and a high failure rate can block users from accessing protected resources. Common causes include:

  • Invalid Credentials: Incorrect or expired credentials used for token generation.
  • Configuration Issues: Misconfigured DataServer or authentication settings.
  • Internal Service Errors: DataServer internal errors or exceptions.
  • Timeouts: Request timeouts due to slow backend services or resource pressure.

Troubleshooting Steps

Step 1: Review Grafana Dashboards

  1. Navigate to Grafana → Dashboards → Application-Dashboards → dataserver → DataServer.
  2. Review the PToken Generation Error Rate panel:
    • This panel shows the percentage of PToken generation requests that failed.
    • Check if the error rate is consistently above 5%.

Step 2: Use Diagnostics Tool

This option is available only for self-managed deployments.

The Diagnostics Tool provides automated testing of DataServer functionality and helps identify configuration or connectivity issues.

  1. Open the Diagnostic Portal and navigate to Dashboard → Pods.
  2. Select the DataServer pod from the available pods list.
  3. Under the CURRENT TEST RESULTS tab, review the PyTest Report for the following checks:
    • test_heathcheck_api: Tests the health check endpoint to ensure the DataServer is responding to requests.
    • test_certificate_api: Validates certificate retrieval functionality.
  4. To check resource utilization, review:
    • test_diag_client_disk_space: Verifies that sufficient disk space is available.
    • test_diag_client_pod_cpu_utilization: Checks CPU usage to identify resource pressure.
    • test_jvm_process_cpu_utilization: Monitors JVM CPU usage for performance bottlenecks.

Step 3: Review DataServer Logs

  1. Download the DataServer logs and search for error patterns. For detailed log collection steps, see the DataServer Service Logs section below.
  2. Look for patterns such as:
    • PToken errors: ptoken, generate_ptoken, token generation
    • Authentication errors: invalid, credentials, authentication, unauthorized

Step 4: Check Configuration

  1. Review the DataServer configuration files for authentication settings.
  2. Verify that credentials and IAM roles are correctly configured.
  3. Check if any recent configuration changes may have caused the issue.

Step 5: Check DataServer Resource Utilization

Resource exhaustion can cause PToken generation requests to fail:

  1. Review Pod Metrics:
    • Navigate to Grafana → Dashboards → Infra-Dashboards → Pod Monitoring.
    • Check pod memory and CPU usage for the DataServer pods.
    • If memory issues are detected, adjust resources using Compute Sizing.

Escalation Checklist

If the issue cannot be resolved through the troubleshooting steps, escalate to Privacera support with the following details. For additional assistance, refer How to Contact Support for detailed guidance on reaching out to the support team.

  • Timestamp of the error: Include the exact time the alert was triggered.
  • Grafana dashboard and alert screenshots:
    1. Grafana → Dashboards → Application-Dashboards → dataserver → DataServer → PToken Generation Error Rate
    2. Grafana → Alerting → Alert rules → PToken Generation Error Rate

  • DataServer Service Logs: Include logs showing PToken generation errors or exceptions.

    Option 1: Download Log from Diagnostic Portal (Recommended)

    This option is available only for self-managed deployments.

    1. Open the Diagnostic Portal and navigate to Dashboard → Pods.
    2. Select the DataServer pod from the available pods list.
    3. Click on the Logs tab and download logs by clicking on DOWNLOAD LOGS button.

    Option 2: Manual Log Collection (If Diagnostic service is not enabled)

    Bash
    1
2
3
4
5
6
7
8
    # Create log archive
kubectl exec -it <POD> -n <NAMESPACE> -- bash -c "cd /workdir/privacera-dataserver/logs/dataserver/ && tar -czf dataserver-logs.tar.gz *.log"

# Copy the archive
kubectl cp <POD>:/workdir/privacera-dataserver/logs/dataserver/dataserver-logs.tar.gz ./dataserver-logs.tar.gz -n <NAMESPACE>

# Extract logs
tar -xzf dataserver-logs.tar.gz