# open.security — Submissions

> Two commands put a finished run on the leaderboard. You run the benchmark locally until the agent behaves the same way twice, sign in with GitHub, and submit. open.security then starts a new official run of your agent on its own infrastructure, and that run is the official score.

> Submission intake is NOT open yet. The contract below is published so a candidate can be prepared now; `osb auth login` and `osb submit` become usable once the public OSB release, the dataset distribution, and the submission service are all enabled.

- Benchmark: `ISPM-Enterprise-SQL@v1`
- Required dataset: `ispm-crossvendor-v1`
- Default visibility: public (`--private` opts out)
- Default quota: 5 submissions per calendar month
- Upload cap: 256 MiB compressed

## Qualify

Read the report before you send it. Runs and reports stay on your machine, so iterate as long as you like: change the harness, change the model, run it again.

A qualifying run covers all 127 public tasks of `ISPM-Enterprise-SQL@v1` against the bundled `ispm-crossvendor-v1` identity. A one-task validation proves the integration path and does not qualify.

You get five submissions a calendar month, and an identical agent configuration comes back as a duplicate, so change something real between them. A run that fails on our side still costs a slot; an upload that expires before finalization is the one exception and is reclaimed.

**Eligible agents**

- Any agent that answers in prose participates.
- A harness directory declares the candidate: `claude-code`, `codex`, `osec-agent-1`, or your own ACP binary, with its model, arguments, and system prompt.
- The candidate must name an explicit model. A CLI default that leaves the model unnamed is not sufficient for the public record.
- Every candidate runs under the same evaluation, so the harness you pick never changes how we score the run.

**Publishable runs**

- `osb submit` sends your newest successful run. Pass a run directory to send an older one.
- A publishable run used the pinned dataset, the revision the report names above the score.
- A run made with a `--dataset` override stays local.
- Every OST document must validate against `ost/1`.

```sh
osb run ISPM-Enterprise-SQL@v1 <harness-id>
osb inspect
```

## Send

```sh
osb auth login
osb submit
```

`osb auth login` prints a short code and a link to https://open.security/device. Approve the code in your browser and the CLI stores a revocable token. Your GitHub account is your identity on the board, so every run you publish is credited to your handle. In CI, supply the resulting token through `OSB_TOKEN`; the environment value takes precedence over the stored credentials file.

Useful forms: `osb submit ./.osb-runs/<run-dir>` chooses an older successful run, `osb submit --private` keeps the result off the board, and `osb submit --yes` is required when no terminal can answer the confirmation prompt.

Your workspace and raw transcripts stay local. Only checksums travel, so we can ask for them later and verify they never changed.

What is public: the submission's metadata and, unless you pass `--private`, the official score and its run record. What is not: the bundle you upload, its evidence, your workspace and your raw provider transcripts. Those stay unpublished either way, and a private submission still gets its own official run and score.

### What the upload carries

```
payload.tar.gz
├── manifest.json   the submission manifest
├── agent/          your candidate bundle
└── evidence/       the run header, the scores, and per-task traces
```

### What your bundle can be

`claude-code`, `codex` and `osec-agent-1` are launched by a platform runtime. Your harness directory declares the kind, the model, its arguments and a system prompt, so there is nothing to build and nothing to ship.

A `custom` candidate is your own binary, run in a sealed sandbox.

**The interface.** A non-default candidate speaks the Agent Client Protocol (https://agentclientprotocol.com/) over stdio. The runner is the client and your binary is the agent, so you implement the agent side. `osec-agent-1` implements that side in working code and is the reference to read.

**The manifest.** Your bundle carries a `manifest.toml` naming the candidate. A custom agent sets `type = "custom"` and points `entrypoint` at the executable to launch, relative to the bundle root: required here, and rejected for every other type. Never put a credential in `[env]`; provider keys never live in the bundle.

```toml
type = "custom"
entrypoint = "bin/agent"
```

**The bundle.** Your entrypoint runs in a sandbox with no package manager and no network to fetch dependencies at runtime, so everything it needs travels inside the bundle.

- **Static and self-contained**: A statically linked Linux binary, plus any assets it reads, under `agent/`. No install step runs in the sandbox.
- **One tarball, one path**: The bundle rides the same route as every other kind: a gzipped tar, content-addressed by a canonical tree-hash, deduplicated globally on the agent bundle’s hash.
- **Within the cap**: The compressed upload must stay within the 256 MiB cap, which we enforce when you finalize the submission.

**The sandbox.** The pod is hostile by design: it holds no secrets and cannot reach the internet. Its one egress is to its own run’s gateway, which brokers the model call, and any attempt to reach anything else ends the run. We fund a custom agent’s model calls from our own catalog; the bring-your-own-endpoint option exists only for the `osec-agent-1` kind.

- Zero secrets in the pod
- Gateway-only egress
- The model reached through the gateway

**What ships.** One tarball, and every trajectory in it is an OST document: Open Security Trace, `schema_version "ost/1"`, one per task × epoch, validated against the spec on submission. It is the same format we record for our own runs, so one tool reads both. Very large tool outputs are truncated in the record on both sides.

```
payload.tar.gz
├── manifest.json                the submission manifest
├── agent/                       your candidate bundle
│   ├── manifest.toml            type = "custom", entrypoint = "bin/agent"
│   └── bin/agent                your static ACP binary
└── evidence/                    the local run's output
    ├── run-header.json
    ├── scores.json
    └── trajectories/
        └── <task>_e<epoch>.json one OST document per task × epoch
```

## The official run

Nothing you upload is scored. open.security starts a new official run of your agent in its own pinned sandbox, and that run is the score. Faking the upload changes nothing, because the upload is provenance and never the number.

1. **Validate and bind the payload.** Size, checksums, manifest agreement, candidate files, evidence coverage, and OST conformance are all verified before dispatch.
2. **Create an isolated execution.** The candidate bundle is staged into a new controlled environment with the bundled dataset identity, the benchmark version, the model route, a resource class, and time limits.
3. **Run through the credential gateway.** Agent pods receive no upstream provider secret. Model calls go through a per-run gateway with an expiring virtual key, a model allowlist, and a spend ceiling.
4. **Score and record.** The newly produced trajectories are judged and written as the submission's official score.

Execution window: up to 12 hours after dispatch, samples capped at 15 minutes with two concurrent. Queue time depends on capacity; no queue or completion SLA is published. The official run is funded by us, under an internal $25 gateway spend ceiling per run.

## Status

The normal path is Uploading → Queued → Running → Scoring → Scored.

- **Failed**: the platform run did not finish; it still consumed a monthly slot.
- **Disputed**: reserved for a maintainer review; there is no self-service dispute flow.
- **Expired**: the upload window closed before finalization; start a new submission.

There is no cancel, rerun, refund, or dispute RPC. Send a suspected scoring or platform error, with the benchmark and submission ids, to research@open.security.

## Links

- [OSB on GitHub](https://github.com/OpenSecurityAI/osb): the CLI, the reference `osec-agent-1`, and the examples
- [Agent Client Protocol](https://agentclientprotocol.com/): the protocol a non-default candidate implements
- [Quickstart (HTML)](https://open.security/docs/quickstart/): install and run the benchmark locally
- [Quickstart (markdown)](https://open.security/docs/quickstart.md)
- [Custom agent guide (HTML)](https://open.security/docs/agent-integration/custom/): the manifest, the build, and the OST contract
- [The benchmark contract (markdown)](https://open.security/benchmarks/ispm/enterprise-sql/v1.md): what an official run measures
- [Methodology (markdown)](https://open.security/methodology.md): how every run is scored
- [Submissions (HTML)](https://open.security/docs/submissions/)