# ISPM-Enterprise-SQL@v1

> A text-to-SQL evaluation over a frozen synthetic enterprise. The agent must query the world, explain the result, and leave a reviewable trace.

## Identity

- Benchmark id: `ISPM-Enterprise-SQL@v1`
- Family: ISPM
- Modality: SQL
- Version: v1
- Status: available
- Bundled dataset: `ispm-crossvendor-v1`
- Public tasks: 127 (75 single-platform, 52 cross-platform)
- Canonical page: https://open.security/benchmarks/ispm/enterprise-sql/v1/

A score is comparable only within this benchmark id, this modality, and the judge configuration each published run records. Scores from a different benchmark, modality, or judge panel are never merged.

## Requirements

- Every task runs against the bundled ispm-crossvendor-v1 SQLite snapshot. Its manifest names the dataset and publish date but does not yet carry a content digest.
- The agent receives the task prompt, SQLite world, and schema documentation. Public ground truth and witness SQL stay outside the agent workspace during the run.
- A response must state the conclusion and identify the people, resources, policies, or configuration facts that support it.
- Official evaluation runs each task for five epochs with two declared, floating judge aliases; the provider-returned wire model IDs are recorded once at preflight.
- Within an epoch the lower non-abstaining judge grade wins. Across epochs, the modal grade wins and ties resolve downward.
- Official execution allows two concurrent samples, 15 minutes per sample, and 12 hours for the complete run; a run timeout produces no partial score.

## Rubric dimensions

Six dimensions, graded per epoch. Within an epoch the lower non-abstaining judge grade wins; across epochs the modal grade wins and ties resolve downward.

### Answer correctness

`answer_correctness`

Semantic agreement with the world-derived ground truth, with partial credit for materially correct but incomplete findings.

### Verdict

`answer_verdict`

For yes-or-no tasks, the primary call must match ground truth. Enumeration tasks do not receive this grade.

### Evidence completeness

`evidence_completeness`

The answer covers the findings needed to support its conclusion, not only a representative example.

### Evidence relevancy

`contextual_evidence_relevancy`

Cited identities and facts bear directly on the prompt and do not substitute adjacent but irrelevant observations.

### Reasoning utility

`reasoning_utility`

The explanation makes the conclusion auditable without rewarding length or a hidden chain of thought.

### SQL semantics

`sql_semantic_appropriateness`

The query meaning fits the schema and question. Structural table and join coverage are also reported deterministically. The evaluator does not require the agent's query to match the witness SQL that derives the world's facts: answer quality and structural query diagnostics are scored separately.

## What an official run publishes

- candidate and resolved model metadata
- aggregate and per-task consensus metrics
- per-epoch judge grades and votes
- token and step accounting
- reliability and consistency statistics

## Links

- [The contract (HTML)](https://open.security/benchmarks/ispm/enterprise-sql/v1/)
- [The task set (markdown)](https://open.security/benchmarks/ispm.md): all 127 tasks with their full taxonomy
- [The dataset (markdown)](https://open.security/datasets/ispm-crossvendor-v1.md): the frozen world these tasks run against
- [Methodology (markdown)](https://open.security/methodology.md): the evaluation every benchmark runs under
- [Leaderboards (markdown)](https://open.security/leaderboards.md): the published standings
- [Submissions (markdown)](https://open.security/docs/submissions.md): how a candidate becomes an official run
