Skip to content
open.securityopen.securityBeta
Skip documentation navigation
Browse documentation

open.security documentation

Run, integrate, and submit

Start with one local task. Then make the candidate reproducible, inspect its evidence, and send the same bundle to open.security for a new official run.
Benchmark
ISPM-Enterprise-SQL@v1
Bundled dataset
ispm-crossvendor-v1

Quickstart

The shortest complete loop ends with one scored task and an inspected local report. It does not create a platform submission.

One task · runtime varies by model latency and retries

Prove the local path before you scale it.

Install the paired binaries, preflight your credentials, run aws-active-access-keys-older, and read the saved result with osb inspect.

Run the quickstart
shell
$ osb doctor
$ osb run ISPM-Enterprise-SQL@v1 default \
  --tasks aws-active-access-keys-older
$ osb inspect

The workflow

Each stage produces something you can inspect before you move on.

  1. Prepare the host

    Install OSB and the matching osec-agent-1 binary from the same archive. Export the judge credentials, then run osb doctor.

  2. Describe one candidate

    A harness/<id>/ directory pins the agent type, model, arguments, non-secret environment, and context. The directory name is the harness id you pass on the command line.

  3. Validate one task

    Run one task first. The preflight checks the manifest, the dataset identity, the provider routes, and the models before the candidate spends a token.

  4. Submit a candidate

    Authenticate once with the browser device flow, send a qualifying full local run, and follow the new official run through scoring.

Focused guides

Choose the runtime that actually executes the candidate. The benchmark, the dataset, and the scoring path do not change with it.

CLI adapter

Claude Code

Use your installed Claude Code CLI, its own authentication, and an explicit headless permission posture.
Open guide

CLI adapter

Codex

Run the installed Codex CLI headlessly with its local login, sandbox policy, and optional reasoning configuration.
Open guide

Reference runtime

osec-agent-1

Use the model-agnostic reference loop and pin the model it should drive.
Open guide

Bring your own binary

Custom ACP agent

Ship a self-contained Linux executable that implements the ACP agent side over stdio.
Open guide

Not sure which fits? Compare all four boundaries.

References

The human-facing references for what is measured, what the world contains, and what becomes public.

Contract

The benchmark

The requirements, the six rubric dimensions, the judge policy, and the evidence an official run publishes.
Read the contract

Scoring

Methodology

How tasks, traces, judges, metrics, and official scores fit together.
Read methodology

Platform

Submission contract

Eligibility, quota, visibility, authentication, and status behaviour before you send.
Prepare a submission

Machine access

The canonical entry points for agents, scripts, and retrieval pipelines. Every content page with a twin also answers to Accept: text/markdown.
/agents.md
Canonical machine overview, workflow, and resource registry.
/llms.txt
Compact discovery index generated from the same resource registry.
/docs/quickstart.md
The local run loop, with every CLI step in a fenced block.
/docs/submissions.md
The submission contract, from the same shared constants.

For local automation, osb doctor --json reports the host preflight without terminal formatting.