Skip to content
open.securityopen.securityBeta
  1. Benchmarks
  2. ISPM
  3. Enterprise SQL v1

ISPM Enterprise SQL

ISPM-Enterprise-SQL@v1

Contract published
A text-to-SQL evaluation over a frozen synthetic enterprise. The agent must query the world, explain the result, and leave a reviewable trace.

What every conforming run holds constant.

These 6 clauses are v1. A change to the task set, the world, the agent-visible surface, the repetition policy or the evidence record is a new revision with a new identifier, not an edit to this one.

Run contract

  1. Every task runs against the bundled ispm-crossvendor-v1 SQLite snapshot. Its manifest names the dataset and publish date but does not yet carry a content digest.

  2. The agent receives the task prompt, SQLite world, and schema documentation. Public ground truth and witness SQL stay outside the agent workspace during the run.

  3. A response must state the conclusion and identify the people, resources, policies, or configuration facts that support it.

  4. Official evaluation runs each task for five epochs with two declared, floating judge aliases; the provider-returned wire model IDs are recorded once at preflight.

  5. Within an epoch the lower non-abstaining judge grade wins. Across epochs, the modal grade wins and ties resolve downward.

  6. Official execution allows two concurrent samples, 15 minutes per sample, and 12 hours for the complete run; a run timeout produces no partial score.

6 judged dimensions, one answer.

A judge grades the response on every dimension below. Structural coverage of the agent’s own SQL is measured deterministically beside them, so a fluent answer cannot erase a weak investigation.

Answer correctness
Semantic agreement with the world-derived ground truth, with partial credit for materially correct but incomplete findings.answer_correctness
Verdict
For yes-or-no tasks, the primary call must match ground truth. Enumeration tasks do not receive this grade.answer_verdict
Evidence completeness
The answer covers the findings needed to support its conclusion, not only a representative example.evidence_completeness
Evidence relevancy
Cited identities and facts bear directly on the prompt and do not substitute adjacent but irrelevant observations.contextual_evidence_relevancy
Reasoning utility
The explanation makes the conclusion auditable without rewarding length or a hidden chain of thought.reasoning_utility
SQL semantics
The query meaning fits the schema and question. Structural table and join coverage are also reported deterministically. The evaluator does not require the agent's query to match the witness SQL that derives the world's facts: answer quality and structural query diagnostics are scored separately.sql_semantic_appropriateness

What an official run publishes.

Public run record

Verified Execution Protocol
  1. 01

    candidate and resolved model metadata

  2. 02

    aggregate and per-task consensus metrics

  3. 03

    per-epoch judge grades and votes

  4. 04

    token and step accounting

  5. 05

    reliability and consistency statistics

Read the parts.

All 127 tasks, filterableThe ispm-crossvendor-v1 datasetThe ISPM-Enterprise-SQL@v1 leaderboardHow a run becomes a score

Run it against this contract.

One binary ships this task set, its pinned dataset and the grader. The score you see locally is the protocol the board runs; a submission is what starts the official run.

Run it locallySubmit a run