Every SRE I have worked with has the same loop. An incident starts. They jump into Grafana. And then they spend the first three minutes not investigating the issue — they spend it remembering the right PromQL query.
The actual problem
The dashboards are usually there. Someone built them years ago for the same kind of incident. But finding the right panel under pressure, reading the query expression, adapting the time window, copying the right label selector — all of this is friction that compounds when the system is on fire.
The usual proposal is "add an AI copilot." Most of these copilots fail in production for the same reason: they hallucinate metric names that do not exist in your environment. A copilot that confidently writes `rate(http_requests_total[5m])` when your team uses `nginx_http_requests_total` is worse than no copilot at all — it adds a wrong answer on top of an existing problem.
The shape of a useful answer
A Retrieval-Augmented Generation (RAG) layer flips the failure mode. Instead of asking the LLM to invent the query, you ask it to find and adapt an existing one.
The knowledge base is everything you already have:
Every panel definition in every Grafana dashboard you own
Every alert rule in Prometheus
Every recorded query in `query_log_archive` or your runbooks
You index those — query expressions, panel titles, descriptions, dashboard tags — into a vector store. The LLM never invents a metric name. It receives the SRE's intent ("API error rate during yesterday's incident"), retrieves the top-N matching panels, and answers with the actual PromQL from those panels, adjusted for the time range mentioned in the request.
Pipeline
User intent (NL)
│
▼
Embedding model (OpenAI text-embedding-3-small or local)
│
▼
Vector search across { panel.title, panel.description, query.expr, dashboard.tags }
│
▼
Top-5 relevant panels with their original PromQL
│
▼
LLM (Claude Haiku is enough — this is structured rewrite, not creative)
│
▼
Final PromQL adapted to the user's time range and labelsThe LLM step is intentionally minimal. It takes structured retrieved context plus the user's question and returns either a PromQL string or a refusal. No "creative" output. No hallucinated metric names — by construction, because the retrieval gave it real ones.
What changes in practice
The SRE asks: "show me the 5xx rate on the checkout API during the deploy at 14:02 BRT yesterday."
The RAG layer retrieves the panel "checkout-api / 5xx by endpoint" from the existing checkout dashboard, adapts the time window to `[2026-05-01T17:02Z, +1h]` (BRT to UTC), and returns:
sum by (endpoint) (
rate(
http_requests_total{service="checkout", status=~"5.."}[5m]
)
)With a link straight to the Grafana panel. No invention, no guesswork — and the SRE keeps the three minutes that incident response actually needs.
Where it breaks (and how to handle it)
The pattern fails gracefully when the knowledge base is sparse. If the user asks about a service that has no dashboards or alert rules indexed, the LLM has nothing to retrieve and should refuse rather than invent. Make this an explicit guardrail in the prompt: "If no relevant panel was retrieved, respond with 'No matching dashboard found' and stop."
The second failure mode is staleness. Dashboards change. Indexing has to run periodically (a daily cron is enough for most teams) and reindex on dashboard updates if you have hooks available.
The third is governance. The retrieval layer reads everything in the Grafana org. If your dashboards expose customer-identifiable labels, the LLM context now includes them. Filter at index time, not at query time.
Why this is better than a copilot
A generic copilot tries to be clever. A RAG over your own dashboards tries to be accurate. For SRE work, accurate beats clever — every time. The system you ship is one that answers with PromQL that already runs on your environment, because it copied it from a panel you already trust.
This is the same logic the rest of AI-augmented DevOps follows: the LLM is the orchestrator, not the source of truth. Your existing operational knowledge is. The LLM just connects the dots faster than your fingers do.
If your operations team is at the point of considering this kind of layer, that is exactly the kind of work Tyber.io engages on. [Get in touch](/contact) — first conversation is exploratory and no-cost.