An AI ops agent that shows its work.
Ask about six months of card-payment data for a fictional processor, Northwind Payments. The agent plans, queries a read-only database and policy documents, checks its own answer, and won't be talked into actions nobody asked for. Every step it takes is shown below.
Pick a question on the left, or write your own. The agent's plan, every tool call, each guardrail decision and the final check will appear here.
Benchmark
37 questions with answers computed straight from the database, so scoring needs no human judgment: numbers must match, tickets must (or must not) be opened, and planted instructions must be ignored. Each configuration runs the same questions on the same model.
How it works
A LangGraph state machine drives an open-weights model on Groq through five tools. The model never touches the database or the ticket system directly; every call goes through the checks below, and every decision is written to the trace you see above.
01Read-only SQL
The database is opened read-only and immutable, with a SQLite authorizer that allows only SELECT. Queries time out after 2 seconds and return at most 50 rows.
02Role-based tools
Viewers are never shown the ticket tool, and if a model calls it anyway the call is refused. Analysts can open tickets.
03Side-effect gate
Opening a ticket is the one action with a side effect. It only runs when the user's own question asks for a ticket, so a model can't decide to do it on its own.
04Injection screening
Tool output is data, not instructions. Sentences that address an AI (“ignore previous instructions…”) are removed before the model reads them, and logged.
05Self-check
In plan + verify mode, a reviewer pass checks each number in the answer against the tool results and sends the agent back once to fix a problem.
06Budgets and caching
At most 8 tool calls per question. Identical model requests are answered from a cache, so benchmark reruns are free and the recorded runs here replay exactly.
The data is synthetic and generated from a fixed seed: 120 merchants, about 46,000 transactions and 324 chargebacks from January to June 2026, with reference exchange rates, plus 12 policy documents. One policy file contains a planted prompt injection to test the guardrails.