Grafana High Availability (HA)¶

This guide explains how to enable Grafana HA in Privacera monitoring. When enabled, Grafana uses a shared database, multiple replicas, HA alerting, a Pod Disruption Budget (PDB), and anti-affinity for resilience.

What Grafana HA gives you¶

Prerequisites¶

• Database: A shared database (PostgreSQL or MySQL) is required for Grafana HA. You can use the same RDBMS service used by Privacera or any other database instance.
• Vars file: You will edit the monitoring vars file. Follow the steps below to set it up.

Setup vars file¶

1. SSH into the instance where Privacera Manager is installed.

2. Navigate to the config directory:

   Bash
   1	cd ~/privacera/privacera-manager/config/

3. Copy vars.monitoring.yml file from sample-vars folder to custom-vars folder:

!!! note If this file already exists in custom-vars folder, you can skip this step.

1

cp sample-vars/vars.monitoring.yml custom-vars/

1. Open the file for editing:

   Bash
   1	vi custom-vars/vars.monitoring.yml

In the rest of this guide, this file is referred to as vars.monitoring.yml.

Step 1: Create the Grafana database and user¶

Before enabling Grafana HA, you must create a dedicated database and user for Grafana in your database server.

Login to your database¶

Connect to your database server using an admin user with privileges to create databases and users.

For MySQL:

1

mysql -u <username> -p -h <hostname> -P 3306

For PostgreSQL:

1

psql -U <username> -d <database_name> -h <hostname> -p 5432

Replace:

• <username> - Admin username
• <hostname> - Database hostname

Create database and user¶

Once logged in, execute the following SQL commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

-- Create database
CREATE DATABASE grafana;

-- Create user (syntax varies by database type)
CREATE USER grafana_user IDENTIFIED BY 'your_secure_password';

-- Grant privileges
GRANT ALL PRIVILEGES ON grafana TO grafana_user;

-- Apply changes
FLUSH PRIVILEGES;

Step 2: Configure Grafana HA variables¶

Edit vars.monitoring.yml and set the following variables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

###############################################
######## Grafana High Availability (HA) #########
###############################################

# Enable Grafana HA (required for all HA features)
GRAFANA_HA_ENABLE: "true"

# --- Database Configuration ---
GRAFANA_DB_TYPE: "mysql"                    # Options: "postgres" or "mysql"
GRAFANA_DB_HOST: "your-db-host.example.com" # Database hostname or IP
GRAFANA_DB_PORT: "3306"                     # Port: 5432 for PostgreSQL, 3306 for MySQL/MariaDB
GRAFANA_DB_NAME: "grafana"                  # Database name you created
GRAFANA_DB_USER: "grafana_user"             # Database user you created
GRAFANA_DB_PASSWORD: "your_secure_password" # Password for the database user

# --- Optional: PostgreSQL only ---
# GRAFANA_DB_SSL_MODE: "require"            # Options: require, verify-full, disable

## Grafana HA Replicas Configuration
## Uncomment to override default replicas (default: 1 when HA enabled)
GRAFANA_DEPLOYMENT_REPLICAS: 1

Variable descriptions¶

What happens when you enable HA¶

When GRAFANA_HA_ENABLE is set to "true", the following are automatically configured:

• Database usage enabled (SQLite disabled)
• Initial replicas set to the value of GRAFANA_DEPLOYMENT_REPLICAS (default: 1).
• HPA enabled with min 1, max 3 replicas
• Deployment strategy changed to RollingUpdate
• Headless service created for HA alerting
• Pod Disruption Budget enabled (min 1 pod available)
• Pod anti-affinity enabled (spread across nodes)
• Persistence disabled (no PVC for Grafana)

Step 3: Optional - Override HPA settings¶

What is HPA?¶

HPA (Horizontal Pod Autoscaler) automatically scales the number of Grafana pods based on resource usage (CPU and memory). When load increases, HPA adds more pods; when load decreases, it removes pods, ensuring optimal resource utilization and performance.

Default HPA values when HA is enabled¶

• Initial replicas: 1 (starting number of Grafana pods)
• HPA min replicas: Set to value of GRAFANA_DEPLOYMENT_REPLICAS
• HPA max replicas: 3 (maximum pods allowed)
• CPU threshold: 70% (scale up when CPU exceeds this)
• Memory threshold: 75% (scale up when memory exceeds this)

HPA variables explained¶

You can override these in vars.monitoring.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Initial number of Grafana pods (starting point before HPA takes over)
GRAFANA_DEPLOYMENT_REPLICAS: 1

# Minimum number of pods HPA will maintain (set to value of GRAFANA_DEPLOYMENT_REPLICAS)
GRAFANA_AUTOSCALING_HPA_MIN_REPLICA: "{{GRAFANA_DEPLOYMENT_REPLICAS}}"

# Maximum number of pods HPA can scale up to (under high load)
GRAFANA_AUTOSCALING_HPA_MAX_REPLICA: 3

# CPU threshold: HPA scales up when average CPU usage exceeds this percentage
GRAFANA_AUTOSCALING_HPA_TARGET_CPU: "70"

# Memory threshold: HPA scales up when average memory usage exceeds this percentage
GRAFANA_AUTOSCALING_HPA_TARGET_MEM: "75"

Step 4: Apply and verify¶

Regenerate and deploy¶

1. Run setup so the Grafana Helm values are regenerated from your vars:

1

./privacera-manager.sh setup

1. Install or upgrade (as you normally do):

1

./pm_with_helm.sh install

Or run your usual playbook that deploys monitoring/Grafana.

Sanity checks¶

• Pods: Check that Grafana pods are running:

  Bash
  1	kubectl get pods -n <namespace> -l app.kubernetes.io/name=grafana

  You should see Grafana pods (e.g. 1 to 3 depending on load and HPA).

• HPA: Verify HPA is active:

  Bash
  1	kubectl get hpa -n <namespace>

  You should see the Grafana HPA with current/min/max replicas.

• DB configuration in generated values: After running setup, verify that database variables are correctly rendered in the Grafana Helm values file. Check the grafana.ini section (database config) in:

  Bash
  1	privacera/privacera-manager/output/kubernetes/helm/monitoring/grafana/grafana-values.yml

  Confirm that the database type, host, port, name, and user are set as per your vars.monitoring.yml. This file is generated from your vars; if DB settings are wrong here, fix them in vars.monitoring.yml and run setup again.

Quick reference: variables summary¶

Required variables for HA¶

Optional variables (override defaults)¶

Automatically configured when HA is enabled¶

• GRAFANA_DB_ENABLED → "true"
• GRAFANA_AUTOSCALING_HPA_ENABLED → "true"
• GRAFANA_POD_DISRUPTION_BUDGET_ENABLED → "true"
• GRAFANA_POD_ANTI_AFFINITY_ENABLED → "true"
• Persistence for Grafana → disabled (no PVC)
• Headless service and HA alerting → enabled

Troubleshooting¶