Skip to content

Customizing Observability Stack

We provide an option to customize various features in Privacera monitoring stack like:

  • Customizing Prometheus (Retention period, PVC size, Taints and Toleration, Node Selectors and Resource Quota)
  • Customizing Grafana (PVC size, Admin login password, Node Selectors and Resource Quota).
  • Creating a custom values file to override the default one for all monitoring component.

To customize the monitoring stack copy vars.monitoing.yml file to custom-vars directory.

  1. SSH into the instance where Privacera Manager is installed.
  2. Navigate to the config directory using the following command:
    Bash
    1
    cd ~/privacera/privacera-manager/config/

  3. Copy vars.monioring.yml file from sample-vars folder to custom-vars folder.

    If this file already exists in the custom-vars folder, you can skip this step.

    Bash
    1
    cp -n sample-vars/vars.monitoring.yml custom-vars/

  4. Open vars.monitoring.yml file.

    Bash
    1
    vi custom-vars/vars.monitoring.yml

Customizing Prometheus Configurations

  • In Prometheus, the default retention period is 15 days. To update the retention period, uncomment the PROMETHEUS_DATA_RETENTION_PERIOD variable in the vars.monitoring.yml file , set it to your desired retention period, and save the changes.
    Bash
    1
    PROMETHEUS_DATA_RETENTION_PERIOD: "30d"
  • The default PVC size in Prometheus is 10Gi in Prometheus.To update the PVC size, uncomment the PROMETHEUS_K8S_PVC_SIZE variable in the vars.monitoring.yml file and and set the desired PVC size.
    Bash
    1
    PROMETHEUS_K8S_PVC_SIZE: "20Gi"
  • If you want to run Prometheus on a specific node, and the desired node is already labeled with the required key and value, uncomment the PROMETHEUS_DEPLOYMENT_NODE_SELECTORS variable in the vars.monitoring.yml file and update the KEY and VALUE with the desired key and value.
    Bash
    1
2
    PROMETHEUS_DEPLOYMENT_NODE_SELECTORS: 
    <KEY>: <VALUE>
  • To update the resource quota of Prometheus, uncomment PROMETHEUS_RESOURCE_MEMORY_LIMIT and PROMETHEUS_RESOURCE_CPU_LIMIT variables in vars.monitoring.yml file and provide the desired values. Default memory limit is 10Gi and CPU is 3 core.
    Bash
    1
2
    PROMETHEUS_RESOURCE_MEMORY_LIMIT: "20Gi"
PROMETHEUS_RESOURCE_CPU_LIMIT: "5"
  • If you want to add taints and tolerations to the Prometheus server, uncomment the variables below and provide the desired values in place of <PLEASE_CHANGE>.
    Bash
    1
2
3
4
5
    PROMETHEUS_TOLERATION_ENABLE: "true"
PROMETHEUS_TOLERATION_KEY: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_VALUE: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_OPERATOR: "<PLEASE_CHANGE>"
PROMETHEUS_TOLERATION_EFFECT : "<PLEASE_CHANGE>"

  • Prometheus Node Exporter runs as a DaemonSet and is responsible for collecting node-level metrics such as CPU, memory, and disk usage, then forwarding them to Prometheus.

    By default, it is disabled. To enable it, uncomment the variable shown below in the vars.monitoring.yml file and save your changes.

    Bash
    1
    PROMETHEUS_NODE_EXPORTER_ENABLED: "true"

  • To disable Prometheus, uncomment the following variable and save the changes:

    Bash
    1
    PROMETHEUS_DEPLOYMENT_ENABLED: "false"
    Note: (Optional) If Prometheus is already deployed, you must also complete the following steps in addition to the one mentioned above.

    The default value of MONITORING_NAMESPACE is privacera-monitoring.

    • Manually uninstall the currently deployed Prometheus Helm chart.
      Bash
      1
2
      helm uninstall prometheus -n <MONITORING_NAMESPACE>
helm uninstall prometheus-blackbox-exporter -n <MONITORING_NAMESPACE>
    • Also, make sure to clean up the Prometheus directory.
      Bash
      1
      rm -rf ~/privacera/privacera-manager/output/kubernetes/helm/<MONITORING_NAMESPACE>/prometheus

Customizing Grafana Configurations

  • The default PVC size in Grafana is 1Gi. To update the PVC size, uncomment the GRAFANA_K8S_PVC_SIZE variable in the vars.monitoring.yml file and set the desired PVC size.
    Bash
    1
    GRAFANA_K8S_PVC_SIZE: "2Gi"
  • If you want to run Grafana on a specific node, and the desired node is already labeled with the required key and value, uncomment the GRAFANA_DEPLOYMENT_NODE_SELECTORS variable in the vars.monitoring.yml file and update the KEY and VALUE with the desired key and value.
    Bash
    1
2
    GRAFANA_DEPLOYMENT_NODE_SELECTORS: 
  <KEY>: <VALUE>
  • To update the default admin password for Grafana, uncomment the GRAFANA_LOGIN_USER_PASSWORD variable in the vars.monitoring.yml file and set it to your desired password.
    Bash
    1
    GRAFANA_LOGIN_USER_PASSWORD: <YOUR_PASSWORD>

Creating custom values file for Monitoring components

If you want to create a custom values file to override the default one for the monitoring components used in Privacera, follow the steps below.

  1. Go to custom-vars folder.

    Bash
    1
    cd ~/privacera/privacera-manager/config/custom-vars

  2. Create the required custom values files for the monitoring components. Choose the file name from the table below. For example, if you want to create the custom values file for Grafana, the file name will be grafana-custom-values.yml

    Note

    You can refer public HELM chart to take the reference for creating custom values file.

    MonitoringComponent Custom values file name Helm Chart Version Public Helm Chart URL
    Grafana grafana-custom-values.yml 9.2.1 Grafana Chart
    Grafana Oncall grafana-oncall-custom-values.yml 1.8.13 Grafana OnCall Chart
    Prometheus prometheus-custom-values.yml 27.16.0 Prometheus Chart
    BlackBox Exporter blackbox-exporter-custom-values.yml 9.7.0 Blackbox Exporter Chart
    Loki loki_custom_values.yml 6.30.1 Loki Chart
    Tempo tempo_distributed_custom_values.yml 1.18.1 Tempo Chart
    Pyroscope pyroscope_custom_values.yml 1.13.4 Pyroscope Chart
    Otel Receiver otel_collector_custom_values.yml 0.126.0 OpenTelemetry Chart
    Otel Scraper otel_scraper_custom_values.yml 0.126.0 OpenTelemetry Chart
    Otel Agent otel_agent_custom_values.yml 0.126.0 OpenTelemetry Chart

  3. Add the necessary values to the custom values file and save it.

  4. Redeploy the Privacera Monitoring components.

Restart Services

After making the configuration changes, you need to restart the services to apply them. To restart the services, follow the steps below.

a. Go to privacera-manager directory.

Bash
1
cd ~/privacera/privacera-manager
b. Run setup to generate the required files.
Bash
1
./privacera-manager.sh setup
c. Install the monitoring components.
Bash
1
./pm_with_helm.sh install-monitoring
d. Run install to update the Grafana.
Bash
1
./pm_with_helm.sh install
e. Once done, run post-install.
Bash
1
./privacera-manager.sh post-install