Skip to content

Ranger Plugin - Audit Spool Not Draining Alert

Root Cause

The Ranger Plugin's audit framework (agents-audit) spools generated audit events to local disk before/while sending them to the configured audit destination(s) (e.g. Solr, Kafka, HDFS). The spool has no directory-size cap — audit events must never be dropped — so if a destination stops draining the spool, disk usage on the plugin pod grows unbounded until the underlying issue is fixed.

A Ranger Plugin Audit Spool Not Draining alert is triggered when the on-disk spool size for a destination has not touched zero at any 1-minute sample across a full 5-minute window. A destination blip that clears within seconds will not trigger this alert — only a spool that stays non-empty for the whole window will.

Common Causes:

  • Destination Unreachable: network problems, DNS failure, or the configured audit destination (Solr, Kafka, HDFS, etc.) being down.
  • Authentication/Authorization Failures: expired or invalid credentials for the audit destination.
  • Destination Overloaded or Slow: the destination is accepting writes far slower than the plugin generates audit events, so the spool backlog keeps growing even though writes are technically succeeding.
  • Misconfiguration: incorrect destination URL, topic, index name, or credentials after a config change or redeploy.
  • Disk Pressure on the Spool Volume: the pod's local disk is low on space, causing spool writes themselves to slow down or fail.

Troubleshooting Steps

Step 1: Identify the Affected Destination

Use the Audit Pipeline (agents-audit) row on the Grafana → Application-Dashboards → plugins → ranger-plugin-audit dashboard. The alert fires per destination label — check the Audit Spool Size and Audit Spool File Count panels, filtered to that destination, to confirm the spool is growing rather than holding steady.

Step 2: Apply Quick Fixes Based on the Likely Cause

Symptom Likely Cause Quick Fix
Destination connection errors in plugin logs Destination unreachable or misconfigured Verify network connectivity and the destination URL/topic/index configuration for that destination
401/403 or auth errors in plugin logs Expired or invalid destination credentials Verify and rotate credentials used by the plugin for that destination
No errors, but spool keeps growing Destination is up but too slow to keep pace Check destination-side load/latency (e.g. Solr/Kafka/HDFS cluster health) and scale or tune it
Spool writes themselves failing/slow Disk pressure on the plugin pod's spool volume Check available disk space on the pod/volume backing the spool directory

Step 3: Confirm Recovery

Once the underlying issue is resolved, the Audit Spool Size and Audit Spool File Count panels for that destination should trend back toward zero. The alert clears once the spool touches zero within a 5-minute window.

Escalation Checklist

If the issue cannot be resolved through standard troubleshooting, escalate it with the following information:

  • Timestamp of the alert and the affected destination label value.
  • Grafana Dashboards and Alert Screenshots:
    • Dashboard Screenshot: Dashboards → Application-Dashboards → plugins → ranger-plugin-audit
    • Alert Screenshot: Dashboards → Application-Dashboards → plugins → Alert rules → Audit Spool Not Draining – Ranger Plugin
  • Ranger Plugin Logs: include any logs showing audit destination connection errors, authentication failures, or write failures around the alert time.