Enabling Tempo¶
Introduction¶
Tempo is a distributed tracing backend designed to ingest and store traces efficiently without requiring indexing. It allows developers to troubleshoot and analyze request flows across microservices. It stores traces in an efficient manner, reducing infrastructure and operational costs. Also, it is designed to handle high volumes of trace data with minimal overhead.
Process¶
- SSH into the instance where Privacera Manager is installed.
- Navigate to the
config
directory using the following command:Bash -
Copy
vars.monioring.yml
file fromsample-vars
folder tocustom-vars
folder.If this file already exists in
custom-vars
folder then you can skip this step.Bash -
Open
vars.monioring.yml
.Bash - Uncomment the below variables in the file and save it.
- Enable Tempo.
Bash
- Enable Tempo.
-
Once done, redeploy the monitoring components.
a. Go to
privacera-manager
directory.b. RunBash setup
to generate the required files.c. Install the monitoring components.Bash d. Once done, runBash post-install
.Bash
Configure Cloud Storage for Tempo¶
This guide explains how to configure cloud-based object storage for Grafana Tempo using AWS S3, Azure Blob Storage, or Google Cloud Storage (GCS) within a production-ready Privacera Monitoring stack.
Configure AWS S3 for Tempo¶
Prerequisites¶
Ensure the following are in place:
- An S3 bucket to store Tempo data.
- An IAM role with the necessary permissions and trust relationship(of kubernetes service account).
Step 1: Ensure that an Identity Provider (IdP) is already created for your EKS cluster’s OIDC. If not, create a new Identity Provider before proceeding.¶
- Navigate to AWS EKS → Select your cluster.
- From the Overview tab, copy the OIDC (OpenID Connect) provider URL.
- Go to IAM → Identity Providers → Add provider.
- Select OpenID Connect.
- Paste the OIDC URL under Provider URL and click Get thumbprint.
- Set Audience as
sts.amazonaws.com
. - Add optional tags and click Add provider. ( we will need this ID to be added in the IAM role.)
Step 2: Create an IAM Policy¶
Go to IAM → Policies, and create a new policy with the following JSON definition:
JSON | |
---|---|
Tip
- Replace
<AWS_S3_BUCKET_NAME>
with your AWS S3 bucket name.
Step 3: Create an IAM Role and Trust Relationship¶
- Navigate to IAM → Roles → Create role.
- Select Web identity as the trusted entity type.
- Choose the OIDC provider created earlier, set Audience to
sts.amazonaws.com
, and proceed. - Attach the custom policy from the previous step.
- Name and create the role.
Once created, modify the trust relationship to limit role assumption to specific Kubernetes service accounts:
Tip
- Replace
<AWS_ACCOUNT_ID>
,& <OIDC_ID>
with your AWS account ID , aws region & OIDC_ID respectively. - Action
sts:AssumeRoleWithWebIdentity
allows a service (like a Kubernetes service account) to assume an IAM role using a web identity token (e.g.OIDC). - Default service account name is
tempo-distributed
and namespace isprivacera-monitoring
.
Step 4: Configure Tempo for S3¶
- SSH into the instance where Privacera Manager is installed.
- Navigate to the configuration directory:
Bash | |
---|---|
- Create the Tempo custom values file:
Bash | |
---|---|
- Add the following configuration:
Tip
- Replace
<AWS_S3_BUCKET_NAME>
,<AWS_REGION>
&<AWS_IAM_ROLE_ARN>
with your S3 bucket name , AWS region & IAM role created above. - Prefix
tempo-data
is directory name which will be created inside aws S3 bucket.
Step 5: Redeploy Monitoring Components¶
a. Go to privacera-manager
directory.
Bash | |
---|---|
setup
to generate the required files. Bash | |
---|---|
Bash | |
---|---|
post-install
. Bash | |
---|---|
Configure Azure Blob Storage for Tempo¶
Step 1: Enable Workload Identity on AKS¶
- Go to Azure Portal → Kubernetes Services → Your AKS Cluster.
- In the left sidebar, go to Settings → Authentication.
- Ensure the following are enabled:
- OIDC Issuer: Enabled
- Workload Identity: Enabled
- Click Save if any changes were made.
Step 2: Create a Storage Account¶
- Go to Azure Portal → Storage Accounts → Create.
- Configure:
- Name: e.g.,
privaceramonitoringsa
(must be globally unique) - Region: Same as your AKS cluster
- Resource Group: Same as AKS
- Performance: Standard
- Replication: LRS
- Name: e.g.,
- Click Review + create, then Create.
Step 3: Create a Blob Container¶
- In your created storage account, go to Data storage → Containers.
- Click + Container and configure:
- Name: e.g.,
privacera-monitoring-container
- Public access level: Private (no anonymous access)
- Name: e.g.,
- Click Create.
Step 4: Create a User-Assigned Managed Identity¶
- Go to Azure Portal → Managed Identities → Create.
- Configure:
- Name: e.g.,
privacera-monitoring-identity
- Region: Same as your AKS cluster
- Resource Group: Same as AKS/storage
- Name: e.g.,
- Click Review + create, then Create.
- After creation, note down:
- Client ID
Step 5: Assign Permissions to the Managed Identity¶
Note
"Make sure you have owner access to your storage account."
- Go to Storage Accounts → Your Storage Account → Access Control (IAM).
- Click + Add → Add role assignment.
- Configure:
- Role: Storage Blob Data Contributor
- Assign access to: Managed identity
- Select member: Choose the
privacera-monitoring-identity
managed identity
- Click Save.
Step 6: Add Federated Credential to the Managed Identity¶
- Go to Azure Portal → Kubernetes Services → Your AKS Cluster → Settings → Authentication.
- Copy the OIDC Issuer URL (e.g.,
https://oidc.prod-aks.azure.com/...
).
Then:
- Go to Azure Portal → Managed Identities → privacera-monitoring-identity → Federated credentials.
- Click + Add credential.
- Fill in the following:
- Name: e.g.,
tempo-federated
- Issuer: Paste the OIDC Issuer URL
- Subject: e.g.,
system:serviceaccount:privacera-monitoring:tempo-distributed
- Audience:
api://AzureADTokenExchange
- Name: e.g.,
- Click Add.
Tip
- Default service account name is
tempo-distributed
and namespace isprivacera-monitoring
.
Step 7: Configure Tempo for Azure¶
- SSH into the Privacera Manager instance.
- Navigate to the config directory:
Bash | |
---|---|
- Create the custom values file:
Bash | |
---|---|
- Add the following content:
Tip
- Replace
<CONTAINER_NAME>
,<STORAGE_ACCOUNT_NAME>
&<AZURE_MANAGED_IDENTITY_CLIENT_ID>
with your azure container name, storage account name & CLINT_ID of managed identity created above.
Step 8: Redeploy Monitoring Components¶
a. Go to privacera-manager
directory.
Bash | |
---|---|
setup
to generate the required files. Bash | |
---|---|
Bash | |
---|---|
post-install
. Bash | |
---|---|
Configure GCP Storage for tempo¶
Step 1: Create a GCS Bucket¶
- Go to GCP Console → Cloud Storage.
- Click Create Bucket.
- Configure:
- Name: Globally unique
- Region: As required
- Storage class: Standard
- Access control: Uniform
- Click Create.
Step 2: Create a Service Account¶
- Go to IAM & Admin → Service Accounts.
- Click Create Service Account.
- Provide a name and (optionally) a description.
- Skip assigning roles for now.
- Click Done.
Step 3: Grant Bucket Access to the Service Account¶
- Go back to your bucket → Permissions tab.
- Click Grant Access.
- Add your service account’s email.
- Assign the following roles:
- Storage Admin
- Storage Object Admin
- Click Save.
Step 4: Configure Workload Identity¶
- Open the service account details.
- Under Permissions, click + Grant Access.
- In New principals, add:
Text Only | |
---|---|
Tip
- Replace
<PROJECT_ID>
with your GCP project id . - Default value for privacera monitoring namespace is
privacera-monitoring
& default k8s service account name istempo-distributed
.
- Assign role:
Service Account Workload Identity User
- Save the configuration.
Step 5: Configure tempo for GCP¶
- SSH into the Privacera Manager instance.
- Navigate to the config directory:
Bash | |
---|---|
- Create the custom values file:
Bash | |
---|---|
- Add the configuration below:
YAML | |
---|---|
Tip
- Replace
<GCP_BUCKET_NAME>
,<GCP_SERVICE_ACCOUNT>
& `with your GCP bucket name, service account name and project id respectively. - Prefix
tempo-data
is directory name which will be created inside GPC bucket.
Step 6: Redeploy Monitoring Components¶
a. Go to privacera-manager
directory.
Bash | |
---|---|
setup
to generate the required files. Bash | |
---|---|
Bash | |
---|---|
post-install
. Bash | |
---|---|