Skip to content

Customizing Observability Stack

We provide an option to customize various features in Privacera monitoring stack like:

  • Customizing Prometheus (Retention period, PVC size, Taints and Toleration, Node Selectors and Resource Quota)
  • Customizing Grafana (PVC size, Admin login password, Node Selectors and Resource Quota).
  • Creating a custom values file to override the default one for all monitoring component.

To customize the monitoring stack copy vars.monitoing.yml file to custom-vars directory.

  1. SSH into the instance where Privacera Manager is installed.
  2. Navigate to the config directory using the following command:
    Bash
    1
    cd ~/privacera/privacera-manager/config/

  3. Copy vars.monioring.yml file from sample-vars folder to custom-vars folder.

    If this file already exists in the custom-vars folder, you can skip this step.

    Bash
    1
    cp -n sample-vars/vars.monitoring.yml custom-vars/

  4. Open vars.monitoring.yml file.

    Bash
    1
    vi custom-vars/vars.monitoring.yml

Customizing Prometheus Configurations

  • In Prometheus, the default retention period is 15 days. To update the retention period, uncomment the PROMETHEUS_DATA_RETENTION_PERIOD variable in the vars.monitoring.yml file , set it to your desired retention period, and save the changes.
    Bash
    1
    PROMETHEUS_DATA_RETENTION_PERIOD: "30d"
  • If you want to run Prometheus on a specific node, and the desired node is already labeled with the required key and value, uncomment the PROMETHEUS_DEPLOYMENT_NODE_SELECTORS variable in the vars.monitoring.yml file and update the KEY and VALUE with the desired key and value.
    Bash
    1
2
    PROMETHEUS_DEPLOYMENT_NODE_SELECTORS: 
    <KEY>: <VALUE>
  • To update the resource quota of Prometheus, uncomment PROMETHEUS_RESOURCE_MEMORY_LIMIT and PROMETHEUS_RESOURCE_CPU_LIMIT variables in vars.monitoring.yml file and provide the desired values. Default memory limit is 10Gi and CPU is 3 core.
    Bash
    1
2
    PROMETHEUS_RESOURCE_MEMORY_LIMIT: "20Gi"
PROMETHEUS_RESOURCE_CPU_LIMIT: "5"
  • If you want to add taints and tolerations to the Prometheus server, uncomment the variables below and provide the desired values in place of <PLEASE_CHANGE>.
    Bash
    1
2
3
4
5
    PROMETHEUS_TOLERATION_ENABLE: "true"
PROMETHEUS_TOLERATION_KEY: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_VALUE: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_OPERATOR: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_EFFECT : "<PLEASE_CHANGE>"

  • Prometheus Node Exporter runs as a DaemonSet and is responsible for collecting node-level metrics such as CPU, memory, and disk usage, then forwarding them to Prometheus.

    By default, it is disabled. To enable it, uncomment the variable shown below in the vars.monitoring.yml file and save your changes.

    Bash
    1
    PROMETHEUS_NODE_EXPORTER_ENABLED: "true"

  • To disable Prometheus, uncomment the following variable and save the changes:

    Bash
    1
    PROMETHEUS_DEPLOYMENT_ENABLED: "false"
    Note: (Optional) If Prometheus is already deployed, you must also complete the following steps in addition to the one mentioned above.

    The default value of MONITORING_NAMESPACE is privacera-monitoring.

    • Manually uninstall the currently deployed Prometheus Helm chart.
      Bash
      1
2
      helm uninstall prometheus -n <MONITORING_NAMESPACE>
helm uninstall prometheus-blackbox-exporter -n <MONITORING_NAMESPACE>
    • Also, make sure to clean up the Prometheus directory.
      Bash
      1
      rm -rf ~/privacera/privacera-manager/output/kubernetes/helm/<MONITORING_NAMESPACE>/prometheus

Increasing Prometheus PVC Size

If you need to increase the Prometheus PVC size after initial deployment, follow the steps below carefully.

Prerequisites

  • Ensure your storage class supports volume expansion (allowVolumeExpansion: true)
  • Verify you have appropriate permissions to patch PVCs

Steps to increase PVC size:

1. Verify storage class supports expansion

Bash
1
kubectl get storageclass <storage-class-name> -o jsonpath='{.allowVolumeExpansion}'

Replace <storage-class-name> with your actual storage class name. The default format is: <K8S_NAMESPACE>-store-privacera-prometheus

Example:

Bash
1
kubectl get storageclass privacera-monitoring-store-privacera-prometheus -o jsonpath='{.allowVolumeExpansion}'

Expected output: true

2. Patch the PVC to increase its size

Bash
1
kubectl patch pvc <pvc-name> -n <monitoring-namespace> -p '{"spec":{"resources":{"requests":{"storage":"<new-size>"}}}}'

Example:

Bash
1
kubectl patch pvc storage-volume-prometheus-server-0 -n privacera-monitoring -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

3. Verify the PVC resize

Check both the requested and actual capacity of the PVC:

Bash
1
2
kubectl get pvc <pvc-name> -n <monitoring-namespace> \
  -o jsonpath='Requested: {.spec.resources.requests.storage}{"\n"}Capacity: {.status.capacity.storage}{"\n"}'

Ensure both Requested and Capacity reflect the new size before proceeding.

4. Copy vars.monitoring.yml to custom-vars

Copy vars.monitoring.yml from the sample-vars folder to the custom-vars folder on the Privacera Manager host if not present.

If this file already exists in the custom-vars folder, you can skip this step.

Bash
1
2
cd ~/privacera/privacera-manager/config
cp -n sample-vars/vars.monitoring.yml custom-vars/

5. Update PROMETHEUS_K8S_PVC_SIZE in vars.monitoring.yml

Critical: Sync values after patching

You must update PROMETHEUS_K8S_PVC_SIZE in vars.monitoring.yml to match the new PVC size before running Helm upgrade. Failing to do so will cause the upgrade to fail.

YAML
1
PROMETHEUS_K8S_PVC_SIZE: "<new-size>"

Example:

YAML
1
PROMETHEUS_K8S_PVC_SIZE: "50Gi"

6. Delete the StatefulSet safely

Critical: Required before Helm upgrade

volumeClaimTemplates in a StatefulSet are immutable — Helm upgrade will fail if the size in vars.monitoring.yml differs from the existing StatefulSet spec. Deleting the StatefulSet with --cascade=orphan is mandatory. This does NOT delete your pods, PVCs, or data.

Find the StatefulSet name:

Bash
1
kubectl get sts -n <monitoring-namespace>

Delete StatefulSet safely — pods and PVCs are preserved:

Bash
1
kubectl delete sts <sts-name> -n <monitoring-namespace> --cascade=orphan

Verify pods are still running after deletion:

Bash
1
kubectl get pods -n <monitoring-namespace>

7. Re-run the Helm upgrade

a. Go to privacera-manager directory.

Bash
1
cd ~/privacera/privacera-manager
b. Run setup to generate the required files.
Bash
1
./privacera-manager.sh setup
c. Install the monitoring components.
Bash
1
./pm_with_helm.sh install-monitoring
d. Run install to update the Grafana.
Bash
1
./pm_with_helm.sh install
e. Once done, run post-install.
Bash
1
./privacera-manager.sh post-install

8. Verify pod is running

Bash
1
kubectl get pod -n <monitoring-namespace> -l app.kubernetes.io/name=prometheus

Ensure prometheus-server-0 is in Running state with 2/2 ready containers.

Customizing Grafana Configurations

  • The default PVC size in Grafana is 1Gi. To update the PVC size, uncomment the GRAFANA_K8S_PVC_SIZE variable in the vars.monitoring.yml file and set the desired PVC size.
    Bash
    1
    GRAFANA_K8S_PVC_SIZE: "2Gi"
  • If you want to run Grafana on a specific node, and the desired node is already labeled with the required key and value, uncomment the GRAFANA_DEPLOYMENT_NODE_SELECTORS variable in the vars.monitoring.yml file and update the KEY and VALUE with the desired key and value.
    Bash
    1
2
    GRAFANA_DEPLOYMENT_NODE_SELECTORS: 
  <KEY>: <VALUE>
  • To update the default admin password for Grafana, uncomment the GRAFANA_LOGIN_USER_PASSWORD variable in the vars.monitoring.yml file and set it to your desired password.
    Bash
    1
    GRAFANA_LOGIN_USER_PASSWORD: <YOUR_PASSWORD>

Creating custom values file for Monitoring components

If you want to create a custom values file to override the default one for the monitoring components used in Privacera, follow the steps below.

  1. Go to custom-vars folder.

    Bash
    1
    cd ~/privacera/privacera-manager/config/custom-vars

  2. Create the required custom values files for the monitoring components. Choose the file name from the table below. For example, if you want to create the custom values file for Grafana, the file name will be grafana-custom-values.yml

    Note

    You can refer public HELM chart to take the reference for creating custom values file.

    MonitoringComponent Custom values file name Helm Chart Version Public Helm Chart URL
    Grafana grafana-custom-values.yml 9.2.1 Grafana Chart
    Grafana Oncall grafana-oncall-custom-values.yml 1.8.13 Grafana OnCall Chart
    Prometheus prometheus-custom-values.yml 27.16.0 Prometheus Chart
    BlackBox Exporter blackbox-exporter-custom-values.yml 9.7.0 Blackbox Exporter Chart
    Loki loki_custom_values.yml 6.30.1 Loki Chart
    Tempo tempo_distributed_custom_values.yml 1.18.1 Tempo Chart
    Pyroscope pyroscope_custom_values.yml 1.13.4 Pyroscope Chart
    Otel Receiver otel_collector_custom_values.yml 0.126.0 OpenTelemetry Chart
    Otel Scraper otel_scraper_custom_values.yml 0.126.0 OpenTelemetry Chart
    Otel Agent otel_agent_custom_values.yml 0.126.0 OpenTelemetry Chart

  3. Add the necessary values to the custom values file and save it.

  4. Redeploy the Privacera Monitoring components.

Restart Services

After making the configuration changes, you need to restart the services to apply them. To restart the services, follow the steps below.

a. Go to privacera-manager directory.

Bash
1
cd ~/privacera/privacera-manager
b. Run setup to generate the required files.
Bash
1
./privacera-manager.sh setup
c. Install the monitoring components.
Bash
1
./pm_with_helm.sh install-monitoring
d. Run install to update the Grafana.
Bash
1
./pm_with_helm.sh install
e. Once done, run post-install.
Bash
1
./privacera-manager.sh post-install