Skip to content
open.securityopen.securityBeta
Skip documentation navigation
Browse documentation

Agent integration

Integrate a custom ACP agent

Bring the agent implementation itself. Your bundle supplies a self-contained Linux executable that speaks ACP over stdio, calls the model through the resolved gateway environment, and leaves an inspectable trajectory for every solve.
Agent kind
custom
Harness id
custom
Benchmark
ISPM-Enterprise-SQL@v1
Bundled dataset
ispm-crossvendor-v1

Prerequisites

Have the runtime and its authentication ready before you author the candidate.
  • OSB and the matching osec-agent-1 binary on PATH.
  • An implementation of the agent side of the Agent Client Protocol.
  • A Linux build environment matching the official runtime. The submitted entrypoint must be statically linked and self-contained.
  • A named model from the platform catalog and provider routes for local validation.
  • No runtime dependency on a package manager, installer, or direct internet access.

Configuration

The manifest is strict. entrypoint is required for custom and rejected for every other agent type. It is relative to the harness root and must resolve to a file in the submitted bundle.

Create harness/custom/manifest.toml:

harness/custom/manifest.toml
type = "custom"
entrypoint = "bin/agent"
 
# Name the catalog model so a submission is reproducible.
model = "anthropic/claude-sonnet-5"

1. Implement the ACP agent side

OSB is the ACP client. Your process reads JSON-RPC 2.0 from stdin, writes JSON-RPC only to stdout, and handles four methods:

initialize
Negotiate the protocol version and identify the agent.
session/new
Open a solve session with its working directory and MCP servers.
session/prompt
Receive the task, run the loop, and stream session updates.
session/cancel
Stop an in-flight turn when the client cancels it.

2. Build the bundle

A Go agent can produce a static Linux entrypoint like this. Validate on Linux so the same file can run locally and in the official environment:

shell
$ mkdir -p harness/custom/bin
$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
  -trimpath -o harness/custom/bin/agent ./cmd/agent
$ chmod 0755 harness/custom/bin/agent
candidate bundle
harness/custom/
├── manifest.toml
├── bin/
│   └── agent
└── <assets read by the agent>
  • Include every runtime asset under the harness directory.
  • Use regular files only; symlinks, FIFOs, and device nodes are refused at submission.
  • Keep the eventual compressed submission under 256 MiB.
  • Never write a credential into [env] or an asset.

3. Honor the sandbox and gateway

A new official run starts your binary in a sealed pod. There is no package manager, no install hook, no provider key in the pod, and no general internet egress. The only model path is the run’s credential gateway.

  • Read OSB_MODEL for the selected model instead of hard-coding a deployment.
  • Use standard provider variables such as OPENAI_BASE_URL and ANTHROPIC_BASE_URL. In the official environment they point to the run’s gateway.
  • The corresponding API-key variable contains a scoped, expiring virtual key, never the upstream provider credential.
  • Do not bypass the supplied base URL. Any other egress attempt ends the run.

4. Produce valid OST evidence

OSB folds the ACP session into one Open Security Trace document per task and epoch. Each document declares schema_version "ost/1" and wraps an ordered record array, validated against the spec on submission.

  • Preserve the distinction between user, reasoning, assistant, and tool records.
  • Match every tool result to a call made by an earlier record.
  • Stream session updates with stable message and tool-call ids.
  • Report usage through the ACP update metadata when your model client exposes it.
  • Treat diagnostics as declared evidence loss, not a place to hide malformed records.
submission payload · produced by osb submit
payload.tar.gz
├── manifest.json                the submission manifest
├── agent/                       your candidate bundle
│   ├── manifest.toml            type = "custom", entrypoint = "bin/agent"
│   └── bin/agent                your static ACP binary
└── evidence/                    the local run's output
    ├── run-header.json
    ├── scores.json
    └── trajectories/
        └── <task>_e<epoch>.json one OST document per task × epoch

Do not assemble this archive yourself. osb submit validates the run and OST documents, snapshots the resolved agent bundle, and builds the canonical payload.

Preflight

Check the agent runtime first, then let OSB check the full host and benchmark configuration.
shell
$ file harness/custom/bin/agent
$ test -x harness/custom/bin/agent
$ osb doctor

Run the one-task validation on Linux with the exact binary you intend to ship. Doctor checks the common host and provider boundaries; command preflight validates the strict custom manifest before the agent starts.

One-task validation

Validate aws-active-access-keys-older before you spend on full coverage.
shell
$ osb run ISPM-Enterprise-SQL@v1 custom \
  --tasks aws-active-access-keys-older

The harness id is the directory name, not an agent flag. OSB resolves the directory, validates the manifest and dataset identity, pings the required models, and only then starts the candidate.

Expected output

The smoke test is complete when the runtime and evidence path are both visible.
  • The setup view identifies the harness type as custom and resolves bin/agent.
  • The ACP initialize and session handshake completes without stdout noise.
  • The agent uses the session’s get_schema and run_query MCP tools.
  • The saved run contains an agent/ snapshot and a conforming trajectory document for the task.
  • The task reaches scored and OSB prints the saved run directory.
shell
$ osb inspect

Troubleshooting

Fix the first failing boundary before changing the model or prompt.
custom requires an entrypoint
Set entrypoint = "bin/agent" and make sure that exact regular file exists under the harness root.
exec format error or missing shared library
Rebuild for the official Linux architecture with static linking. Validate the final binary on Linux before creating a full run.
ACP JSON decode fails
Keep stdout exclusively for JSON-RPC and redirect every logger and SDK diagnostic to stderr.
Tool results are absent from the trajectory
Return ACP session updates with stable tool-call ids and preserve the result that answers each call. OSB cannot bind an orphan result after the fact.
Model calls work locally but fail officially
Remove hard-coded vendor URLs and direct credentials. Read the supplied model, base URL, and key variables so calls stay inside the per-run gateway.
Submission rejects a trajectory
Inspect the named OST document. Fix the ACP event stream that produced it rather than editing evidence after the run.

Next

Move from a working smoke test to a stable, complete candidate.

Compare

Choose another agent

Return to the integration hub and compare runtime boundaries.
Compare agents

Run

Complete coverage

Run the full benchmark against the bundled dataset and inspect the result.
Review the run loop

Publish

Submit the candidate

Check eligibility, quota, visibility, authentication, and platform status.
Prepare a submission