Skip to content
open.securityopen.securityBeta
Skip documentation navigation
Browse documentation

Quickstart

Run your first benchmark task

One concrete outcome: a successful local run of aws-active-access-keys-older, saved on your machine and opened with osb inspect. The OSB CLI is a single Go binary that runs the whole benchmark locally — every task, both judges, and the report. Your agent and the judges call the model providers you configure; nothing reaches open.security until you sign in.
Benchmark
ISPM-Enterprise-SQL@v1
Bundled dataset
ispm-crossvendor-v1
Task
aws-active-access-keys-older
Quickstart harness
default

Prepare

A local run executes a candidate on your host. Start in a clean project directory and decide what isolation that host needs.
  • macOS or Linux, on amd64 or arm64.
  • Provider credentials for both declared judge aliases.
  • Credentials or a local login for whichever agent path you choose.
  • A sandbox you control, if the candidate’s permission posture allows broad host access. The CLI adds no isolation of its own.

Install the paired release

The archive carries osb and the matching osec-agent-1, with the benchmark and its pinned dataset embedded. The commands below document that artifact; they are not a claim that an anonymous install can complete today.

RELEASE BINARIES — NOT YET PUBLISHED

Stage 0 · spec published

The OSB source and release repository opens before public runs are enabled. Until then this page documents the workflow. The organization on GitHub is where the release will appear.

shell
$ tar -xzf osb_<version>_<os>_<arch>.tar.gz
$ mkdir -p ~/.local/bin
$ install -m 0755 osb osec-agent-1 ~/.local/bin/
$ export PATH="$HOME/.local/bin:$PATH"
$ osb version

Add ~/.local/binto your shell profile once the command works. A mismatched reference-agent binary can invalidate the run’s tool surface, so upgrade both files from the same archive.

Export the credentials

Credentials live in the inherited environment or in a machine-local routing config. Never put a provider key in a harness manifest.

Two kinds, and they are not the same key

Judge seats. The ranked board pins one panel: an Anthropic model and an OpenAI model, from different providers. Every run intended for the board therefore reads ANTHROPIC_API_KEY and OPENAI_API_KEY. Scores are comparable only within that fixed pair.

The candidate’s own model, whatever its harness declares. The builtin default harness runs DeepSeek-V4-Pro through an OpenAI-compatible endpoint, so it wants OPENAI_API_KEY plus OPENAI_BASE_URL. A private run does not have to use the board’s panel; a run for the board does.

  1. Export the provider credentials

    shell
    $ export ANTHROPIC_API_KEY=...
    $ export OPENAI_API_KEY=...
    $ export OPENAI_BASE_URL=https://your-compatible-endpoint/v1

  2. Preflight before spending tokens

    shell
    $ osb doctor

    Doctor checks the paired binaries, the agent CLIs, the credentials, the model routing, and the embedded benchmark configuration. Fix every required failure before continuing.

Choose the agent path

The command below drives the builtin default harness. To evaluate another candidate, configure its harness directory first and pass that directory name instead.

Uses your CLI login

Claude Code

Choose this when Claude Code is already installed and you can state a safe headless permission posture.
Configure Claude Code

Uses your CLI login

Codex

Choose this when Codex is installed and its local sandbox and authentication are already understood.
Configure Codex

Reference loop

osec-agent-1

Choose this to pin a model behind the small, inspectable reference agent loop.
Configure osec-agent-1

Your executable

Custom ACP agent

Choose this when you own an ACP implementation and can ship it as a self-contained Linux bundle.
Configure a custom agent

Run one task

Keep the first validation deliberately small. The benchmark still applies its five-epoch policy and both declared judge aliases to the task you name.
shell
$ osb run ISPM-Enterprise-SQL@v1 default \
  --tasks aws-active-access-keys-older
  • Preflight finishes before the candidate starts spending tokens.
  • The task moves from queued through running and judging to scored.
  • OSB prints a run directory under ./.osb-runs/.

Inspect the saved result

With no path, inspect opens the newest saved run. Pass the printed run directory to reopen an older one.
shell
$ osb inspect
$ osb inspect --json > result.json

Confirm four things

  • The run header names ISPM-Enterprise-SQL@v1 and ispm-crossvendor-v1.
  • The harness and resolved agent type are the candidate you intended.
  • The run completed without partial samples.
  • The task report carries the judge votes, the usage, and the final task score.

What a local run costs and where it goes

Five things worth knowing before the full task set, each of which a reader checks one of.
Iterating
Runs and reports stay on your machine, so iterate as long as you like. --tasks trims a run to the tasks you name while you tune.
Isolation
The CLI adds none of its own, so run it inside your own sandbox.
What leaves
Signing in and submitting are the only commands that talk to us. Everything else stays between your machine and your model providers.
Cost
Your agent’s tokens plus both judges’, across 127 tasks. The official run is on us.
How long
A few hours end to end for the full task set. Run a single task first to check the wiring.

Next steps

Keep the same inspectable loop as you move from plumbing to a real candidate.

Configure

Integrate your agent

Create the harness directory that makes the candidate reproducible.
Choose an agent

Publish

Prepare a submission

Run complete coverage, check eligibility, authenticate, and send the candidate.
Read the contract

Understand

Read the methodology

What the local report measures, before you compare two candidates.
Read methodology

When the candidate is ready

shell
$ osb run ISPM-Enterprise-SQL@v1 default
$ osb inspect

Replace default with your own harness id. Complete coverage of all 127tasks against the bundled dataset is required before a submission qualifies. Submit once the agent behaves the same way twice — stability, not a flattering number, is what the board’s own tiebreak measures.