Four commitments before any score.
The evaluator is inspectable, repeatable, public about uncertainty, and versioned when the contract changes.
Accuracy/ Sets the rank
P—01Ground truth
Every answer is checked against a closed set of entities drawn from the environment’s own data: exactly these, no more, no fewer. Naming none of them and inventing one are both wrong, so a score is checkable against something.
P—02Uncertainty first
Every number ships with the sample size behind it, and a bootstrap 95% confidence interval wherever there are at least two observations to resample. A single observation gets no interval at all. A small lead reads as a small lead.
P—03The full record
Official runs publish the score and everything behind it: what the agent answered, how it reasoned, and every query it ran against the snapshot.
P—04Open to inspection
The questions, the ground truth, the judge rubrics, the CLI and the papers are all public, so anyone can audit or reproduce a result. The reference SQL and the generator stay unpublished, and rotating the environment stops a published dataset being answered from memory. Neither guards against overfitting to a schema and a task set we publish in full. Held-out splits would, and there are none yet.
The agent gets two tools and no answers: one returns the schema, the other runs one read-only query at a time.
Nothing is handed over as files. The enterprise is served through get_schema and run_query, against a disposable per-task copy, so the harness records every statement and every row it returns. The surface is the same whether the candidate is our reference agent, Claude Code or Codex.
75 of the 127 questions are answerable inside a single system: a password policy, a root account, a set of stale keys. 52 require resolving the same person across systems that declare no keys between them, and 11 of those span three platforms or more. The schema tool hands over column names and row counts, never types and never foreign keys, so the agent has to work out for itself which columns refer to the same person.
- AWS
- Azure
- BambooHR
- Google Workspace
- Okta
What we measure.
M—01/ Sets the rank
Accuracy
The mean judged answer score across every task in the benchmark, reported with its 95% confidence interval. Ties on the board break by consistency. It is not a weighted composite of the rubric dimensions: the rank comes from answer correctness alone, and the other judged dimensions are published as diagnostics beside it.
M—02
Consistency
How steadily results repeat across the five epochs every task runs, measured on the ranking metric. pass@1 is the mean over tasks of the fraction of that task’s scored epochs that came out fully right. pass^5 estimates that all five attempts succeed, over the tasks with all five epochs scored; a task with an unscored epoch leaves that estimate rather than counting against it. The rank uses pass^5 to break ties.
M—03
Table recall
On this board every task is answered by querying the enterprise. We extract the tables and joins from the agent’s own SQL and compare both against the reference. The board column is table recall: whether the agent reached the data the answer depends on. Join fidelity, whether it reconstructed the relationships between systems, ships in the full report with precision and F1 for both.
M—04
Answer and query quality
Of the tasks that pose a yes/no question, the share where the agent’s final call matches ground truth: binary under the panel’s consensus, with no partial credit. Enumeration tasks, the ones whose answer is a set of entities rather than a call, carry no verdict and are excluded, so this is not a share of the whole benchmark. A separate judged metric scores the semantic appropriateness of the agent’s SQL against the schema, kept in the full report.
M—05
Efficiency
What the run cost in billed tokens (input, cache, and output). The board shows the total across the full run; the report breaks it out for the agent and the judges separately.
M—06
Judge agreement
The two judges score every epoch independently, and their agreement ships as unweighted Cohen’s kappa, computed per metric and then averaged. High agreement says the panel is consistent, not that it is right. Their pooled votes set each task’s grade: the mode across all five epochs, with ties resolving to the lower grade, so no single epoch decides a grade on its own.
Five steps, one loop.
Every run moves through the same five steps, and the last one hands back to the first.
What we publish per run.
Every official run publishes all 6 entries below, whatever the score.
run_query with the rows that came back.What this does not measure
mfa_required is a per-application flag here; production Okta uses sign-on policies.Read the inputs, then read the results.
The task set is what every agent is asked. The board is how they answered. Both are open to read now.