Skip to content

Alert Notification

Trust3AI's alert rules live in the Trust3 Managed folder, in per-application subfolders (for example Infra dashboards, or an Omni Connector title for connector rules), alongside the dashboards, and each carries a severity label (critical or warn) you can route on. The rules ship with no notification settings, so you decide where alerts go by configuring two things in your plane's Grafana:

  • A contact point: the destination (Slack, PagerDuty, Microsoft Teams, webhook).
  • A notification policy: the routing that sends an alert to the right contact point, usually by its severity label.

This works for the Trust3-managed rules and any rules you add yourself.

Prerequisites

  • A Self-Managed runtime plane, and the admin sign-in for its Grafana. The Grafana URL and default credentials are shown under System Applications → Grafana in the plane's console; on non-EKS planes (no URL there) reach Grafana with kubectl port-forward svc/grafana 3000:80 -n <runtime-plane-name>.
  • Destination details: for Slack / Microsoft Teams / PagerDuty / webhook, the webhook URL, API token, or integration key.

Set your own Grafana admin password

The default is the runtime plane name, shown on the console. Because contact points store live secrets (Slack tokens, webhook keys), change it to your own in Application Configs on the plane console (adminPassword) before adding them. A change made inside Grafana's profile resets on pod restart.

Email is not available

The plane's Grafana has no SMTP server configured, so email contact points cannot send. Use Slack, Microsoft Teams, PagerDuty, or a webhook.

Add your own alert rule

Besides the Trust3-managed rules, you can add your own. Grafana evaluates your query on a schedule and fires when the condition is met.

  1. Go to Alerting → Alert rules and select + New alert rule.
  2. Under Enter alert rule name, name the rule.
  3. Under Define query and alert condition, pick the Prometheus data source, build the query, and set the Alert condition (for example, WHEN QUERY IS ABOVE a threshold).
  4. Under Add folder and labels, add a lowercase severity label (critical or warn) so the rule routes through your notification policies.
  5. Under Set evaluation behavior, choose an evaluation group and a Pending period (how long the condition must hold before firing).
  6. Under Configure notifications, select a Contact point, or leave routing to your notification policy. See Step 1: Create a contact point and Step 2: Configure a notification policy.
  7. (Optional) Under Configure notification message, add a Summary and Description.
  8. Select Save.

Grafana reference: Create alert rules · Set up Grafana Alerting

Step 1: Create a contact point

A contact point is where alerts are delivered.

  1. Go to Alerting → Notification configuration and open the Contact points tab.
  2. Select + New contact point.
  3. Fill in the fields described below.
  4. Select Test to confirm it works, then Save contact point.
Field What to enter
Name A unique name for this contact point.
Integration The channel type (Slack, PagerDuty, Microsoft Teams, Webhook, etc.); the other fields follow from it. See the supported integrations.
Integration settings The connection details, for example the webhook URL or channel for Slack.
Optional settings Title/message overrides and whether to notify on resolved alerts. To apply a notification template (Slack): under Optional Slack settings → Edit the message / text body, select the template trust3ai.alert.message.template, and save. The trust3ai.alert.message.template renders the alert's Summary (not its Description), so put the text responders need in Summary.

Build your own notification template (optional)

To create your own template, see Grafana's Create a notification template and Associate a notification template with a contact point.

Example A: Webhook contact point

  • Name: Ops Webhook
  • Integration: Webhook
  • URL: your endpoint, for example https://alerts.yourcompany.com/grafana

Example B: Slack contact point

Pick one method:

Example: Name Critical Slack, Integration Slack, then the webhook URL or token + recipient.

Grafana reference: Manage contact points

Step 2: Configure a notification policy

A notification policy routes firing alerts to your contact points. It lives under Alerting → Notification configuration → Notification policies, with one default policy (a catch-all) plus routes you add under it. An alert is delivered by the first route whose labels match, or by the default policy if none match.

Option A: Point the default policy at one destination

  1. Open the Notification policies tab.
  2. On the Default policy, select More → Edit.
  3. Set Default contact point to a contact point from Step 1.
  4. (Optional) Set Group by and the Group wait / Group interval / Repeat interval.
  5. Select Update policy.

The default policy ships pointing at a no-op receiver (empty) that delivers nowhere, so set its contact point to a real destination; otherwise any alert that matches no route is silently dropped. Select Test on that contact point to confirm it delivers.

Option B: Route by severity

Send each severity to a different contact point by adding one route per severity.

  1. On the Notification policies tab, select + Add route.
  2. Under Matching labels, set Label severity, Operator =, and Value critical.
  3. Set the Contact point for this route (for example, Critical Slack).
  4. (Optional) Turn on Override grouping and set Group by; turn on Continue matching subsequent sibling nodes to let the alert also match later routes.
  5. (Optional) Turn on Override general timings and set Group wait / Group interval / Repeat interval.
  6. Select Add route. Repeat for each severity (for example, a warn route to another contact point).

Set the default contact point too

Option B only adds routes; the default (catch-all) policy still points at the no-op empty receiver. Set the default policy's contact point as in Option A, or any alert that matches no route (a rule with no severity label, or a typo like warning) is silently dropped.

Example: route by severity

  • severity = critical → Critical Slack (a Slack contact point)
  • severity = warn → Ops Webhook
  • Default policy → Ops Webhook (catch-all)

Grafana reference: Manage notification policies

Step 3: Verify

  1. Under Alerting → Alert rules, confirm the rule's state (Normal, Pending, Firing).
  2. When it fires (or via Test on the contact point), confirm the notification arrives.

Copy a managed alert rule for editing

The Trust3-managed rules are provisioned by Trust3AI: you can edit one in Grafana, but your edit is overwritten on the next update (a deleted rule is recreated; a pause is kept). To change the alerting itself, copy the rule, edit the copy, and pause the original:

  1. Open the rule → More → Duplicate.
  2. Give it a new name and save it to your own folder / evaluation group, outside Trust3 Managed.
  3. Save; the copy gets a new random UID, so Trust3AI doesn't manage it.
  4. Pause the original managed rule, so that only your copy fires.
  5. Edit the copy freely; your edits persist across updates.