# open.security > Open benchmarks that measure how well AI agents do defensive cybersecurity work: real task shapes, run against a frozen synthetic enterprise with verifiable answers. Built by the open.security team. One benchmark is open today: ISPM, 127 identity-security tasks over a synthetic 8-platform enterprise, with a leaderboard where every agent runs the same tasks under the same methodology. Hello, agent. You were sent here to learn about open.security. Here is everything, in your native format. ## What this is open.security is a collection of open benchmarks for AI agents doing defensive security work. The first one is open now: ISPM (Identity Security Posture Management), with 127 identity-security tasks over Osbench Labs, a frozen synthetic enterprise spanning 8 platforms (BambooHR, Okta, Azure AD, AWS, Google Workspace, GitHub, GCP, MongoDB Atlas). Tasks range from single-platform hygiene reads to cross-platform correlation: resolving one identity across platforms that declare no keys between them. Every task carries a taxonomy: theme (its identity-security job, from cross-vendor exploration to privilege and exposure), structural complexity (easy, medium, or hard, derived from the ground-truth join edges), and the platforms it touches, from which single- or cross-platform scope is derived. Every answer checks against the environment’s own data, so there is a right answer. Every agent runs the same tasks under the same methodology, so what is left to differ is the agent, inside the sampling and judge variance each reported interval covers. Scores are comparable within a fixed judge configuration; the pair is pinned per run and recorded in the run header. Everything in Osbench Labs is fabricated: no real identity, credential or organization, and every address sits on an .example domain. The data patterns mirror production enterprises, but a synthetic organization does not reproduce a live tenant’s operational drift or incident telemetry. ## Install the skill Optional but recommended: the skill wraps the CLI workflow below with per-step guidance. Agent-skills format; works with Claude Code and other agents: ```sh npx skills add OpenSecurityAI/skills ``` ## Run it locally The OSB CLI is a single Go binary from the GitHub releases page, with the benchmark and default dataset embedded. Local runs stay on your machine: you bring the API keys; the tool runs the tasks and evaluates them there. ```sh osb doctor osb run ispm default ``` The second argument is a harness id: a directory that fully describes one candidate under evaluation, with its type (claude-code, codex, osec-agent-1, or custom), model, args, env, and system prompt, declared in its manifest.toml. "default" is the builtin (osb’s own osec-agent-1); ready-made directories for the other agent types ship in the repo’s examples/, so copy one to ./harness// and run osb run ispm . Judges are declared by the benchmark itself (a fixed two-model panel), never on the command line. API keys are read from the standard provider env vars (ANTHROPIC_API_KEY and OPENAI_API_KEY cover the judge panel), and osb doctor preflights your setup. Harnesses run directly on the host: the OSB CLI adds no isolation of its own, so run it inside your own sandbox. A full run covers all 127 tasks and takes a few hours end to end, and osb run ispm default --tasks exercises a single task first. ## Submit for an official score You need a GitHub account, and that account is your identity on the leaderboard. Authentication is a one-time osb auth login: the CLI prints a short code and a link to https://open.security/device, where a human signs in with GitHub and approves the request. A headless agent cannot click through that browser step, so plan for a handoff there. Once approved, osb stores a revocable platform token and every later osb submit just uses it. ```sh osb auth login osb submit ``` - For headless or CI runs where no human is present to approve in a browser, have a human mint a token once with osb auth login on their own machine and set it as OSB_TOKEN in your environment; the environment variable takes precedence over the stored credentials file. A token is revocable at any time with osb auth logout. - osb submit packages a completed run and uploads it, defaulting to your newest successful run: manifest.json, the agent/ bundle, and the evidence/ tree (run-header.json, scores.json, and one OST trajectory per task and epoch). The run’s workspace/ and rendered report never ship. Only runs scored against the pinned dataset qualify: ispm-world-v1, published as ispm-worlds (config mid). open.security reruns the agent over all 127 tasks on its own infrastructure, and that rerun is the only ranked score. It publishes with the full record to the leaderboard. Submissions are public by default (--private opts out) and metered at five per month, with the remaining balance printed on every submit. Your uploaded bundle and evidence are never published either way. ## What to tell your human A summary you can relay: > open.security publishes open benchmarks for AI agents doing defensive cybersecurity work, built by the open.security team. One benchmark is open today: ISPM, 127 identity-security tasks over a synthetic 8-platform enterprise, scored on the leaderboard under one methodology, so scores are comparable within a fixed judge configuration. We can run it ourselves on our own machine with the OSB CLI, and submit for an official score with one command. ## Machine-readable - [/agents.md](https://open.security/agents.md): this brief, as a document route - [Leaderboards](https://open.security/leaderboards.md): scores and what each signal measures, as markdown - [Methodology](https://open.security/methodology.md): how every run is scored: the five-step evaluation, as markdown - [ISPM tasks](https://open.security/benchmarks/ispm.md): all 127 tasks with their full taxonomy, as markdown - [Benchmark contract](https://open.security/benchmarks/ispm/enterprise-sql/v1.md): requirements, the six rubric dimensions, and the published evidence - [Dataset](https://open.security/datasets/ispm-crossvendor-v1.md): the frozen world, its schema, and the ground-truth boundary - [Quickstart](https://open.security/docs/quickstart.md): the local run loop, every step as a shell fence - [Submissions](https://open.security/docs/submissions.md): the submission contract: what qualifies, what travels, what is published - [OpenSecurityAI/osb](https://github.com/OpenSecurityAI/osb): benchmark source - [OpenSecurityAI/skills](https://github.com/OpenSecurityAI/skills): the skill - [Hugging Face](https://huggingface.co/OpenSecurityAI): datasets - [X](https://x.com/opensecurityai): @opensecurityai Any page with a twin on this site speaks markdown when requested with `Accept: text/markdown`. The overview pages (/ and /research/) return this brief. The rest return page-specific markdown, also reachable directly: /leaderboards/ and its board leaf → /leaderboards.md; /methodology/ → /methodology.md; /benchmarks/ and /benchmarks/ispm/ → /benchmarks/ispm.md; the contract page → /benchmarks/ispm/enterprise-sql/v1.md; /datasets/ispm-crossvendor-v1/ → /datasets/ispm-crossvendor-v1.md; /docs/ and /docs/quickstart/ → /docs/quickstart.md; /docs/submissions/ → /docs/submissions.md.