Choose the agent you want to evaluate

OSB evaluates one reproducible candidate directory at a time. The agent type decides which executable starts; the benchmark, dataset, task, evidence, and scoring contracts do not change with it.

Benchmark: ISPM-Enterprise-SQL@v1
Bundled dataset: ispm-crossvendor-v1

Choose by ownership boundary

The first three use a known runtime. Custom makes your binary part of the submitted candidate.

Installed CLI

Claude Code

Best when you already operate Claude Code and want to carry CLAUDE.md, skills, or subagents into the candidate.

Configure Claude Code

Installed CLI

Codex

Best when you already operate Codex and want its native login, configuration, and workspace-write sandbox.

Configure Codex

Reference runtime

osec-agent-1

Best for a small reference loop where the model is explicit and OSB owns the agent implementation.

Configure osec-agent-1

Submitted binary

Custom ACP agent

Best when the agent itself is your product and you can implement ACP, package a static binary, and honour the gateway boundary.

Configure a custom agent

Agent integration tradeoffs
Agent	What you provide	Model	Primary risk to check
Claude Code	CLI install, login, permission posture	CLI default or manifest pin	A headless permission prompt stalls the task
Codex	CLI install and local login/config	CLI default or manifest pin	System-prompt override is unsupported
osec-agent-1	Matching release binary and model route	Required in the manifest	Model name and provider route must agree
Custom	Static Linux ACP executable and assets	Platform catalog through the gateway	stdout must remain a clean JSON-RPC channel

The common contract

Every path follows the same directory, preflight, one-task validation, and inspection loop.

Create a harness directory

The directory name is the harness id you pass on the command line. Its contents are the candidate, not a loose set of flags.

candidate layout

harness/<id>/
├── manifest.toml          # candidate identity; commit this
├── config.toml            # optional machine-local routing
├── CLAUDE.md | AGENTS.md  # optional context; choose one
├── skills/                # Claude Code only
└── agents/                # Claude Code only

Keep identity separate from machine routing
manifest.toml declares the candidate. An optional config.toml holds local concurrency, dataset, and provider-routing detail. Gitignore that machine-local file, and never write a secret into either.
03
Run preflight
shell
$ osb doctor
Doctor reports host readiness. The strict manifest loader runs again in the command preflight and rejects unknown keys, credential fields, and invalid per-agent combinations.

Validate exactly one task

shell

$ osb run ISPM-Enterprise-SQL@v1 <harness-id> \
  --tasks aws-active-access-keys-older
$ osb inspect

One protocol underneath

Every kind is driven as an Agent Client Protocol agent over stdio. The CLI choices use adapters; osec-agent-1 and custom implement the agent side directly.

OSB is the ACP client and opens one session per solve.
The benchmark data surface arrives as MCP tools, including get_schema and run_query.
Session updates become the ordered evidence saved with the run.
Local model credentials stay in your environment; platform credentials stay behind the per-run gateway.

What a good one-task validation proves

Treat the smoke test as an integration gate, not a score claim.

The intended executable starts without an interactive prompt.
The candidate receives the benchmark MCP tools over ACP.
The provider routes respond before sustained token spend.
The task reaches scored and the run can be inspected.
The saved header names the intended harness, the agent type, ISPM-Enterprise-SQL@v1, and ispm-crossvendor-v1.