Skip to content
open.securityopen.securityBeta
Skip documentation navigation
Browse documentation

Submissions

Prepare an official submission

Public submission intake is not open yet. This contract defines how a reproducible candidate will be authenticated, uploaded, run, scored, and published when it opens, so you can prepare one now.
Benchmark
ISPM-Enterprise-SQL@v1
Required dataset
ispm-crossvendor-v1
Submission intake
release pending
Default visibility
public
Default quota
5 per calendar month
Upload cap
256 MiB compressed

INTAKE CLOSED — CONTRACT PUBLISHED

Stage 0 · spec published

Use the qualification requirements below to prepare a candidate today. osb auth login and osb submit become usable only once the public OSB release, the dataset distribution, and the submission service are all enabled.

1. Qualify the local run

Submission is a reproducibility gate. The CLI refuses a run that cannot produce a complete, comparable candidate bundle.
  • The run completed successfully with full public task coverage — all 127 tasks of ISPM-Enterprise-SQL@v1.
  • The header names the bundled ispm-crossvendor-v1 identity. That manifest is name-bound, not digest-bound; a run made with a --dataset override stays local.
  • The run carries the resolved agent/ snapshot, run-header.json, scores.json, and per-task trajectory evidence.
  • Every applicable metric was scored and every OST document validates against ost/1.
  • The candidate names an explicit model. A CLI default that leaves the model unnamed is not sufficient for the public record.
  • The payload contains regular files only and stays below the 256 MiB compressed cap.
shell
$ osb run ISPM-Enterprise-SQL@v1 <harness-id>
$ osb inspect

2. Choose visibility with the quota in mind

Visibility changes who can read the result. It does not create a free evaluation path.

Default

Public

osb submit. When scoring completes, the official metadata, the score, and a public-safe run record can appear on the leaderboard and at its canonical run URL.

Optional

Private

osb submit --private. The candidate still receives a full official evaluation, but the result stays off the leaderboard and is reachable only through an authenticated run record.

3. Authenticate this machine

The CLI uses open.security's browser device flow. GitHub establishes the submitter identity, and your handle is your identity on the board; the benchmark data plane never uses the browser session.

  1. Start device authentication

    shell
    $ osb auth login

  2. Approve the short code

    The CLI prints a code and opens https://open.security/device on an interactive terminal. Sign in with GitHub and approve the named device.

  3. Store the platform token

    After approval the CLI mints a revocable osb_... token and stores it with owner-only permissions, keyed by platform endpoint.

Headless or CI

A human has to complete the browser step once. Supply the resulting token through OSB_TOKEN in CI; the environment value takes precedence over the stored credentials file. Revoke and remove a stored token with osb auth logout.

4. Inspect, then send

With no run directory, submit chooses the newest successful run and names any newer incomplete run it skipped.
shell
$ osb inspect
$ osb submit

The confirmation names the benchmark, the harness, the model, the judge panel, the epoch count, the dataset, the headline metrics, and the billed tokens. Read it before approving the send.

Useful forms

shell
$ osb submit ./.osb-runs/<run-dir>
$ osb submit --private
$ osb submit --yes

Pass a run directory to choose an older successful run. --yes is required when no terminal is available to answer the confirmation prompt.

5. What travels, and what a bundle must be

The upload is provenance. It is never the score, so what it has to be is complete and verifiable rather than flattering.

What the upload carries

Your workspace and raw transcripts stay local. Only checksums travel, so we can ask for them later and verify they never changed.

one submission
payload.tar.gz
├── manifest.json   the submission manifest
├── agent/          your candidate bundle
└── evidence/       the run header, the scores, and per-task traces

A custom bundle, in full

claude-code, codex and osec-agent-1 are launched by a platform runtime, so there is nothing to build and nothing to ship. A custom candidate is your own binary, and it speaks the Agent Client Protocol over stdio — the runner is the client, your binary is the agent. See the custom-agent guide for the manifest and the build.

Static and self-contained
A statically linked Linux binary, plus any assets it reads, under agent/. No install step runs in the sandbox.
One tarball, one path
The bundle rides the same route as every other kind: a gzipped tar, content-addressed by a canonical tree-hash, deduplicated globally on the agent bundle’s hash.
Within the cap
The compressed upload must stay within the 256 MiB cap, which we enforce when you finalize the submission.

The sandbox

The pod is hostile by design: it holds no secrets and cannot reach the internet. Its one egress is to its own run’s gateway, which brokers the model call, and any attempt to reach anything else ends the run. We fund a custom agent’s model calls from our own catalog; the bring-your-own-endpoint option exists only for the osec-agent-1 kind.

  • Zero secrets in the pod
  • Gateway-only egress
  • The model reached through the gateway

The record format

One tarball, and every trajectory in it is an OST document: Open Security Trace, schema_version "ost/1", one per task × epoch, validated against the spec on submission. It is the same format we record for our own runs, so one tool reads both. Very large tool outputs are truncated in the record on both sides.

submission payload · produced by osb submit
payload.tar.gz
├── manifest.json                the submission manifest
├── agent/                       your candidate bundle
│   ├── manifest.toml            type = "custom", entrypoint = "bin/agent"
│   └── bin/agent                your static ACP binary
└── evidence/                    the local run's output
    ├── run-header.json
    ├── scores.json
    └── trajectories/
        └── <task>_e<epoch>.json one OST document per task × epoch

6. A new run starts on open.security

Nothing you upload is scored. open.security starts a new official run of your agent in its own pinned sandbox, and that run is the score. Faking the upload changes nothing, because the upload is provenance and never the number.

  1. Validate and bind the payload

    The platform verifies size, checksums, manifest agreement, candidate files, evidence coverage, and OST conformance before dispatch.

  2. Create an isolated execution

    open.security stages the candidate bundle into a new controlled environment with the bundled dataset identity, the benchmark version, the model route, a resource class, and time limits.

  3. Run through the credential gateway

    Agent pods receive no upstream provider secret. Model calls go through a per-run gateway with an expiring virtual key, a model allowlist, and a spend ceiling.

  4. Score and record the official result

    The platform judges the newly produced trajectories, records the available evidence-binding state, and writes the result as the submission’s official score.

7. Publication and private results

Visibility applies to the official platform record, not to every artifact involved in producing it.

What is public: the submission's metadata and, unless you pass --private, the official score and its run record. What is not: the bundle you upload, its evidence, your workspace and your raw provider transcripts. Those stay unpublished either way, and a private submission still gets its own official run and score.

Publication behaviour for public and private submissions
ArtifactPublic submissionPrivate submission
Official score and public-safe run recordPublished after scoringOwner-only
Submitter handle and candidate metadataPublishedOwner-only
Agent bundle, evidence, raw trajectoriesNever publicNever public
Workspace and raw provider transcriptsNever uploadedNever uploaded

8. Time, funding, and review

An official run uses shared capacity. The limits below are operational constraints, not a service-level promise.

Execution window

Up to 12 hours after dispatch

Samples are capped at 15 minutes and two run concurrently. Queue time depends on available capacity; no queue or completion SLA is published.

Current funding

The official run is on us

The official path uses platform catalog models under an internal $25 gateway spend ceiling per run. Submitters are not billed today; the monthly quota is what rations shared capacity.

9. Status and troubleshooting

The normal path is Uploading → Queued → Running → Scoring → Scored. A terminal exception says whether you can act.
Uploading
The CLI is creating and finalizing the payload.
Queued
The submission is waiting for an execution slot.
Running
The new official run is active.
Scoring
Execution finished and the official evidence is being scored.
Scored
The official result is complete.
Failed
The platform run did not finish; read the status guidance before sending again.
Disputed
Reserved for a maintainer review; there is no self-service dispute flow.
Expired
The upload window closed before finalization; start a new submission.
Not authenticated, or the token was revoked
Run osb auth login again, or replace the CI OSB_TOKEN.
Duplicate submission
That agent bundle already exists. Change the candidate itself, validate a fresh complete local run, and submit that new bundle.
Monthly quota reached
Wait for the next calendar month, or use an account-specific override if one has been granted. Switching to private does not avoid quota.
Custom dataset is not comparable
Create a full run against the bundled ispm-crossvendor-v1 identity, with no --dataset override.
The run is incomplete or missing evidence
Create a fresh full local run with the current OSB release. Do not hand-edit scores or trajectory files.
The upload window expired
Run osb submit again to create a new submission and upload URL. The expired orphan is reclaimed.
The official run failed
Confirm the same bundle completes locally, then send it again. The failed run still consumed a monthly slot. If platform failures repeat, report the submission id; sensitive infrastructure detail stays redacted.