# open.security — Quickstart

> One concrete outcome: a successful local run of `aws-active-access-keys-older`, saved on your machine and opened with `osb inspect`. The OSB CLI is a single Go binary that runs the whole benchmark locally — every task, both judges, and the report. Your agent and the judges call the model providers you configure; nothing reaches open.security until you sign in.

> RELEASE BINARIES — NOT YET PUBLISHED. The OSB source and release repository opens before public runs are enabled. Until then this document describes the workflow, not an availability claim.

- Benchmark: `ISPM-Enterprise-SQL@v1`
- Bundled dataset: `ispm-crossvendor-v1`
- Validation task: `aws-active-access-keys-older`
- Quickstart harness: `default` (builtin)

## Prepare

- macOS or Linux, on amd64 or arm64.
- Provider credentials for both declared judge aliases.
- Credentials or a local login for whichever agent path you choose.
- A sandbox you control, if the candidate's permission posture allows broad host access. The CLI adds no isolation of its own.

Running and inspecting never contact open.security. The candidate and the judges do contact the model providers you configure. `osb auth login` and `osb submit` are the only commands that talk to us.

## Install the paired release

The archive carries `osb` and the matching `osec-agent-1`, with the benchmark and its pinned dataset embedded. NOTE: release binaries are not published yet, so the commands below document the artifact rather than a path an anonymous install can complete today.

```sh
tar -xzf osb_<version>_<os>_<arch>.tar.gz
mkdir -p ~/.local/bin
install -m 0755 osb osec-agent-1 ~/.local/bin/
export PATH="$HOME/.local/bin:$PATH"
osb version
```

A mismatched reference-agent binary can invalidate the run's tool surface, so upgrade both files from the same archive.

## Export the credentials

Two kinds, and they are not the same key.

**Judge seats.** The ranked board pins one panel: an Anthropic model and an OpenAI model, from different providers. Every run intended for the board therefore reads `ANTHROPIC_API_KEY` and `OPENAI_API_KEY`. Scores are comparable only within that fixed pair.

**The candidate's own model,** whatever its harness declares. The builtin `default` harness runs DeepSeek-V4-Pro through an OpenAI-compatible endpoint, so it wants `OPENAI_API_KEY` plus `OPENAI_BASE_URL`.

Credentials live in the inherited environment or in a machine-local routing config. Never put a provider key in a harness manifest.

```sh
export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...
export OPENAI_BASE_URL=https://your-compatible-endpoint/v1
osb doctor
```

Doctor checks the paired binaries, the agent CLIs, the credentials, the model routing, and the embedded benchmark configuration. Fix every required failure before continuing. `osb doctor --json` is the same check without terminal formatting.

## Choose the agent path

The command below drives the builtin `default` harness. To evaluate another candidate, configure its harness directory first and pass that directory name instead.

- `claude-code` — https://open.security/docs/agent-integration/claude-code/
- `codex` — https://open.security/docs/agent-integration/codex/
- `osec-agent-1` — https://open.security/docs/agent-integration/osec-agent-1/
- `custom` — https://open.security/docs/agent-integration/custom/

With no harness yet, run bare `osb`: it stages an authoring skill and launches your coding agent to write the harness directory with you.

## Run one task

```sh
osb run ISPM-Enterprise-SQL@v1 default \
  --tasks aws-active-access-keys-older
```

- Preflight finishes before the candidate starts spending tokens.
- The task moves from queued through running and judging to scored.
- OSB prints a run directory under `./.osb-runs/`.

A one-task local run is a wiring test. It is neither complete benchmark coverage nor a platform result, and it cannot be submitted as the official record.

## Inspect the saved result

```sh
osb inspect
osb inspect --json > result.json
```

Confirm four things: the run header names `ISPM-Enterprise-SQL@v1` and `ispm-crossvendor-v1`; the harness and resolved agent type are the candidate you intended; the run completed without partial samples; and the task report carries the judge votes, the usage, and the final task score.

## What a local run costs

- **Iterating**: runs and reports stay on your machine. `--tasks` trims a run to the tasks you name.
- **Isolation**: the CLI adds none of its own, so run it inside your own sandbox.
- **What leaves**: signing in and submitting are the only commands that talk to us.
- **Cost**: your agent's tokens plus both judges', across 127 tasks. The official run is on us.
- **How long**: a few hours end to end for the full task set. Run a single task first to check the wiring.

## When the candidate is ready

```sh
osb run ISPM-Enterprise-SQL@v1 default
osb inspect
```

Replace `default` with your own harness id. Complete coverage of all 127 tasks against the bundled dataset is required before a submission qualifies. Submit once the agent behaves the same way twice — stability, not a flattering number, is what the board's own tiebreak measures.

## Links

- [Quickstart (HTML)](https://open.security/docs/quickstart/)
- [Submissions (markdown)](https://open.security/docs/submissions.md): the submission contract
- [The benchmark contract (markdown)](https://open.security/benchmarks/ispm/enterprise-sql/v1.md): what an official run measures
- [The dataset (markdown)](https://open.security/datasets/ispm-crossvendor-v1.md): the frozen world and its ground-truth boundary
- [Methodology (markdown)](https://open.security/methodology.md): how every run is scored