Skip to content
open.securityopen.securityBeta
Skip documentation navigation
Browse documentation

Agent integration

Integrate Claude Code

Drive your installed Claude Code CLI headlessly over ACP. Its local login remains yours; the harness pins the candidate’s model, arguments, context, and permission posture.
Agent kind
claude-code
Harness id
claude-code
Benchmark
ISPM-Enterprise-SQL@v1
Bundled dataset
ispm-crossvendor-v1

Prerequisites

Have the runtime and its authentication ready before you author the candidate.
  • OSB and the matching osec-agent-1 binary on PATH.
  • The claude CLI installed and authenticated; confirm an interactive session can start.
  • Provider credentials and routes for the benchmark’s judge panel.
  • A deliberate headless permission policy. Use your own outer sandbox if the policy grants broad autonomy.

Configuration

Claude Code uses its own local authentication. The harness never stores that login or a provider key.

Create harness/claude-code/manifest.toml:

harness/claude-code/manifest.toml
type = "claude-code"
 
# Optional: omit to use the CLI default.
# model = "claude-sonnet-5"
 
# Headless Claude needs an explicit permission posture.
args = ["--dangerously-skip-permissions"]

Context and assets

  • Put candidate context in harness/claude-code/CLAUDE.md.
  • Optional skills/ and agents/trees are staged into Claude’s native directories.
  • A [system_prompt] block may append or override, with exactly one of file or content.
  • Put only non-secret values in [env]; credential variable names are rejected.

Preflight

Check the agent runtime first, then let OSB check the full host and benchmark configuration.
shell
$ claude --version
$ osb doctor

Doctor must find claude, the paired OSB binaries, and every judge route. Manifest validation then catches unknown keys and missing prompt files before a run starts.

One-task validation

Validate aws-active-access-keys-older before you spend on full coverage.
shell
$ osb run ISPM-Enterprise-SQL@v1 claude-code \
  --tasks aws-active-access-keys-older

The harness id is the directory name, not an agent flag. OSB resolves the directory, validates the manifest and dataset identity, pings the required models, and only then starts the candidate.

Expected output

The smoke test is complete when the runtime and evidence path are both visible.
  • The setup view identifies the harness type as claude-code.
  • Claude starts without waiting on an interactive permission prompt.
  • The session receives get_schema and run_query through the benchmark MCP server.
  • The task reaches scored and OSB prints the saved run directory.
shell
$ osb inspect

Troubleshooting

Fix the first failing boundary before changing the model or prompt.
claude is not found
Install the CLI and make sure claude --version succeeds in the same shell that starts OSB.
The task waits forever at the first turn
Headless Claude is likely waiting for permission. Declare a supported posture in args; do not rely on an interactive prompt.
A skill or subagent is missing
Keep it under the harness skills/ or agents/tree and confirm the run’s saved agent/ snapshot contains it.
Provider preflight fails
Claude’s login and the judge panel are separate boundaries. Fix the provider route named by osb doctor without putting its key in the manifest.

Next

Move from a working smoke test to a stable, complete candidate.

Compare

Choose another agent

Return to the integration hub and compare runtime boundaries.
Compare agents

Run

Complete coverage

Run the full benchmark against the bundled dataset and inspect the result.
Review the run loop

Publish

Submit the candidate

Check eligibility, quota, visibility, authentication, and platform status.
Prepare a submission