Skip to content

Customizing Observability Stack

We provide an option to customize various features in Privacera monitoring stack like:

  • Customizing Prometheus (Retention period, PVC size, Taints and Toleration, Node Selectors and Resource Quota)
  • Customizing Grafana (PVC size, Admin login password, Node Selectors and Resource Quota).
  • Configuring S3 bucket for Loki, Tempo and Pyroscope for data retention (Only applicable for AWS).
  • Creating a custom values file to override the default one for all monitoring component.

To customize the monitoring stack copy vars.monitoing.yml file to custom-vars directory.

  1. SSH into the instance where Privacera Manager is installed.
  2. Navigate to the config directory using the following command:
    Bash
    1
    cd ~/privacera/privacera-manager/config/

  3. Copy vars.monioring.yml file from sample-vars folder to custom-vars folder.

    If this file already exists in the custom-vars folder, you can skip this step.

    Bash
    1
    cp -n sample-vars/vars.monitoring.yml custom-vars/

  4. Open vars.monitoring.yml file.

    Bash
    1
    vi custom-vars/vars.monitoring.yml

Customizing Prometheus Configurations

  • In Prometheus, the default retention period is 15 days. To update the retention period, uncomment the PROMETHEUS_DATA_RETENTION_PERIOD variable in the vars.monitoring.yml file , set it to your desired retention period, and save the changes.
    Bash
    1
    PROMETHEUS_DATA_RETENTION_PERIOD: "30d"
  • The default PVC size in Prometheus is 10Gi in Prometheus.To update the PVC size, uncomment the PROMETHEUS_K8S_PVC_SIZE variable in the vars.monitoring.yml file and and set the desired PVC size.
    Bash
    1
    PROMETHEUS_K8S_PVC_SIZE: "20Gi"
  • If you want to run Prometheus on a specific node, and the desired node is already labeled with the required key and value, uncomment the PROMETHEUS_DEPLOYMENT_NODE_SELECTORS variable in the vars.monitoring.yml file and update the KEY and VALUE with the desired key and value.
    Bash
    1
2
    PROMETHEUS_DEPLOYMENT_NODE_SELECTORS: 
    <KEY>: <VALUE>
  • To update the resource quota of Prometheus, uncomment PROMETHEUS_RESOURCE_MEMORY_LIMIT and PROMETHEUS_RESOURCE_CPU_LIMIT variables in vars.monitoring.yml file and provide the desired values. Default memory limit is 10Gi and CPU is 3 core.
    Bash
    1
2
    PROMETHEUS_RESOURCE_MEMORY_LIMIT: "20Gi"
PROMETHEUS_RESOURCE_CPU_LIMIT: "5"
  • If you want to add taints and tolerations to the Prometheus server, uncomment the variables below and provide the desired values in place of <PLEASE_CHANGE>.
    Bash
    1
2
3
4
5
    PROMETHEUS_TOLERATION_ENABLE: "true"
PROMETHEUS_TOLERATION_KEY: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_VALUE: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_OPERATOR: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_EFFECT : "<PLEASE_CHANGE>"

Customizing Grafana Configurations

  • The default PVC size in Grafana is 1Gi. To update the PVC size, uncomment the GRAFANA_K8S_PVC_SIZE variable in the vars.monitoring.yml file and set the desired PVC size.
    Bash
    1
    GRAFANA_K8S_PVC_SIZE: "2Gi"
  • If you want to run Grafana on a specific node, and the desired node is already labeled with the required key and value, uncomment the GRAFANA_DEPLOYMENT_NODE_SELECTORS variable in the vars.monitoring.yml file and update the KEY and VALUE with the desired key and value.
    Bash
    1
2
    GRAFANA_DEPLOYMENT_NODE_SELECTORS: 
  <KEY>: <VALUE>
  • To update the default admin password for Grafana, uncomment the GRAFANA_LOGIN_USER_PASSWORD variable in the vars.monitoring.yml file and set it to your desired password.
    Bash
    1
    GRAFANA_LOGIN_USER_PASSWORD: <YOUR_PASSWORD>

Configuring S3 Bucket

If you are on AWS, you must provide the S3 bucket name for data retention of Loki, Pyroscope, and Tempo. To do so, follow the steps below.

  • In vars.monitoring.yml file, uncomment MONITORING_S3_BUCKET_NAME and add the S3 Bucket name which you want to use and save it.
    Bash
    1
    MONITORING_S3_BUCKET_NAME: "<S3_BUCKET_NAME>"

Creating custom values file for Monitoring components

If you want to create a custom values file to override the default one for the monitoring components used in Privacera, follow the steps below.

  1. Go to custom-vars folder.

    Bash
    1
    cd ~/privacera/privacera-manager/config/custom-vars

  2. Create the required custom values files for the monitoring components. Choose the file name from the table below. For example, if you want to create the custom values file for Grafana, the file name will be grafana-custom-values.yml

    Note

    You can refer public HELM chart to take the reference for creating custom values file.

    MonitoringComponent Custom values file name Helm Chart Version
    Grafana grafana-custom-values.yml 8.5.1
    Grafana Oncall grafana-oncall-custom-values.yml 1.8.13
    Prometheus prometheus-custom-values.yml 25.8.2
    BlackBox Exporter blackbox-exporter-custom-values.yml 8.9.0
    Loki loki_custom_values.yml 0.79.3
    Tempo tempo_distributed_custom_values.yml 1.18.1
    Pyroscope pyroscope_custom_values.yml 1.7.1
    Otel Receiver otel_collector_custom_values.yml 0.80.1
    Otel Scraper otel_scraper_custom_values.yml 0.80.1
    Otel Agent otel_agent_custom_values.yml 0.80.1

  3. Add the necessary values to the custom values file and save it.

  4. Redeploy the Privacera Monitoring components.

Restart Services

After making the configuration changes, you need to restart the services to apply them. To restart the services, follow the steps below.

Bash
1
2
3
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade

Comments