Skip to content
open.securityopen.securityBeta
Skip documentation navigation
Browse documentation

Agent integration

Integrate osec-agent-1

Use open.security’s model-agnostic reference loop as the candidate. The runtime is fixed and inspectable; the manifest must name the model it should drive.
Agent kind
osec-agent-1
Harness id
osec-agent-1
Benchmark
ISPM-Enterprise-SQL@v1
Bundled dataset
ispm-crossvendor-v1

Prerequisites

Have the runtime and its authentication ready before you author the candidate.
  • OSB and osec-agent-1 installed from the same release archive.
  • A model name supported by your provider route. Unlike the CLI adapters, the reference agent has no model default.
  • The candidate model credential plus credentials for every judge seat required by the benchmark.
  • A compatible base URL when the model is served outside a first-party endpoint.

Configuration

The model is part of candidate identity. Routing is machine-local and may come from standard provider environment variables or an optional config.toml.

Create harness/osec-agent-1/manifest.toml:

harness/osec-agent-1/manifest.toml
type = "osec-agent-1"
 
# Required: there is no runtime default.
model = "claude-sonnet-5"

Model routing

Bare model names are routed by shape: claude* to Anthropic, gpt*, codex*, and o-series names to OpenAI, and gemini* to Gemini. Prefix an ambiguous model explicitly, for example openai/DeepSeek-V4-Pro.

harness/osec-agent-1/config.toml · optional
[llm.anthropic]
base_url = "https://your-endpoint/anthropic"
api_key_env = "MY_PROVIDER_KEY"

api_key_env is the name of an environment variable, never the credential itself. Gitignore this machine-local file.

Reference-agent boundary

  • The agent receives the benchmark MCP tools and has no shell or staged dataset.
  • It emits ACP session updates that OSB records into the run evidence.
  • Its model calls use the provider route OSB resolved for the candidate.

Preflight

Check the agent runtime first, then let OSB check the full host and benchmark configuration.
shell
$ osb version
$ osb doctor

Doctor checks that the sibling reference binary is present, the configured model route resolves, and the benchmark’s judge routes are ready. Do not launch osec-agent-1 directly: its stdout is an ACP channel and it waits for an ACP client.

One-task validation

Validate aws-active-access-keys-older before you spend on full coverage.
shell
$ osb run ISPM-Enterprise-SQL@v1 osec-agent-1 \
  --tasks aws-active-access-keys-older

The harness id is the directory name, not an agent flag. OSB resolves the directory, validates the manifest and dataset identity, pings the required models, and only then starts the candidate.

Expected output

The smoke test is complete when the runtime and evidence path are both visible.
  • The setup view identifies the harness type as osec-agent-1 and prints the pinned model.
  • The reference agent completes its ACP handshake with OSB.
  • Its session calls get_schema and run_query through the provided MCP server.
  • The task reaches scored and OSB prints the saved run directory.
shell
$ osb inspect

Troubleshooting

Fix the first failing boundary before changing the model or prompt.
manifest requires a model
Add an explicit model value. This is the only bundled kind without a CLI-level default.
Provider cannot be inferred
Prefix the model with its API dialect, such as anthropic/model-name or openai/model-name.
The model endpoint returns not found
Keep the canonical model in the manifest and map it to the deployment name under [llm.<provider>.models] in the machine-local config.
ACP handshake or tool surface fails
Reinstall osb and osec-agent-1 from the same archive. A stale sibling binary is not a valid fallback.
Judge preflight fails
The candidate model route and the benchmark’s declared judge aliases are separate. Supply every provider route named by osb doctor.

Next

Move from a working smoke test to a stable, complete candidate.

Compare

Choose another agent

Return to the integration hub and compare runtime boundaries.
Compare agents

Run

Complete coverage

Run the full benchmark against the bundled dataset and inspect the result.
Review the run loop

Publish

Submit the candidate

Check eligibility, quota, visibility, authentication, and platform status.
Prepare a submission