# open.security: ISPM benchmark

> Identity Security Posture Management: 127 identity-security questions over Osbench Labs, a frozen synthetic enterprise spanning eight platforms. Every task is classified by theme, scope, and structural complexity, over the platforms it touches, and every answer comes from the environment’s own data, so there is a right answer to check against.

The tasks run from single-platform hygiene reads to the hard part: resolving one identity across platforms that declare no keys between them, then following the join paths that turn scattered records into a single person.

Platforms: BambooHR, Okta, Azure AD, AWS, Google Workspace, GitHub, GCP, MongoDB Atlas.

Papers:
- [Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibility](https://arxiv.org/abs/2601.07880)
- [Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoning](https://arxiv.org/abs/2606.02674)

## Taxonomy

Every task carries these dimensions:

- **theme**: what the question is about, i.e. the identity-security job it performs: auth-posture, credential-hygiene, cross-vendor-exploration, dormant-stale, inventory-visibility, lifecycle-offboarding, or privilege-exposure
- **platforms**: the vendors involved (aws, azure, gcp, github, gws, bamboohr, mongo, okta); a task can span several
- **scope**: single-platform or cross-platform (derived from the platforms)
- **systems spanned**: 1, 2, or 3+ platforms correlated (derived from the platforms)
- **complexity**: easy, medium, or hard, derived from the count of ground-truth join edges: none, one, and two or more

## All 127 tasks

| id | question | theme | scope | systems spanned | complexity | platforms |
| --- | --- | --- | --- | --- | --- | --- |
| aws-access-keys-were-created | Are there users with access keys that were created during setup but never used? | dormant-stale | single-platform | 1 | medium | aws |
| aws-active-access-keys-creation | Do any active access keys have a creation date older than 90 days? | dormant-stale | single-platform | 1 | easy | aws |
| aws-active-access-keys-older | Which active access keys are older than 90 days? | dormant-stale | single-platform | 1 | easy | aws |
| aws-awssupportaccess-managed-policy-attached | Is the AWSSupportAccess AWS managed policy attached to at least one principal (user, group, or role)? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-belong-common-group | Are there users who do not belong to the common IAM group? | inventory-visibility | single-platform | 1 | medium | aws |
| aws-both-console-password-one | Do any IAM users have both a console password and one or more active API keys? | credential-hygiene | single-platform | 1 | medium | aws |
| aws-console-passwords-but-mfa | Which IAM users have console passwords enabled but no MFA? | auth-posture | single-platform | 1 | easy | aws |
| aws-customer-managed-policies-allow-specific | Do any customer-managed policies allow a specific service:action on all resources (*)? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-customer-managed-policies-attachments | Are there customer-managed policies with no attachments? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-customer-managed-policies-grant-permissions | Do any attached customer-managed policies grant permissions other than sts:AssumeRole? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-customer-managed-policies-use-notaction | Do any customer-managed IAM policies use NotAction in Allow statements? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-ec2-service-roles-attached | Are there EC2 service roles that are not attached to any running EC2 instances? | privilege-exposure | single-platform | 1 | medium | aws |
| aws-groups-inline-policies-attached | Do any IAM groups have inline policies attached? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-hardware-mfa-device-root | Is a hardware MFA device enabled on the root account? | auth-posture | single-platform | 1 | easy | aws |
| aws-high-privilege-non-human-identities-type | List high-privilege non-human identities by type (AWS execution roles, AWS cross-account roles). | inventory-visibility | single-platform | 1 | easy | aws |
| aws-human-policies-attached-directly | Do any human IAM users have policies attached directly (inline or managed)? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-inline-managed-policies-attached | Are there users with inline or managed policies attached directly? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-inline-policies-allow-actions | Do any user inline policies allow actions other than sts:AssumeRole? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-inline-policies-identities-use | Do any inline policies in IAM identities use NotAction with an Allow effect? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-managed-policies-allow-action | Are there any managed policies that allow Action = "*" and Resource = "*"? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-mfa-root | Is MFA enabled on the root account? | auth-posture | single-platform | 1 | easy | aws |
| aws-one-active-access-key | Do any users have more than one active access key? | auth-posture | single-platform | 1 | easy | aws |
| aws-password-enabled-mfa | Do all password-enabled IAM users have MFA enabled? | auth-posture | single-platform | 1 | easy | aws |
| aws-password-expiration-policy | Is password expiration enabled in the IAM password policy? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-password-policy-enforce-expiration | Does the account password policy enforce password expiration and set MaxPasswordAge ≤ 90 days? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-password-policy-prevent-reuse | Does the password policy prevent the reuse of previous passwords? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-password-policy-require-least | Is the IAM password policy configured to require at least 14 characters? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-password-policy-require-least-2 | Does the IAM password policy require at least one lowercase letter? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-password-policy-require-least-3 | Does the password policy require at least one numeric character? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-password-policy-require-least-4 | Does the password policy require at least one special character? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-password-policy-require-least-5 | Does the IAM password policy require at least one uppercase letter? | credential-hygiene | single-platform | 1 | easy | aws |
| aws-passwords-access-keys-used | Are there any users with passwords or access keys that have not been used in over 90 days and are still enabled? | dormant-stale | single-platform | 1 | medium | aws |
| aws-policies-deny-actions-group | Do any policies deny user actions on group resources rather than on the users themselves? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-roles-allow-sts-assumerole | Do any IAM roles allow an AWS principal to perform the sts:AssumeRole action without requiring MFA in their trust policies? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-roles-inline-policies-allow | Do any IAM roles have inline policies that allow a specific service:action on all resources (*)? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-roles-inline-policies-attached | Do any IAM roles have inline policies attached? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-roles-inline-policies-attached-2 | Which roles have inline policies attached? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-roles-trust-external-accounts | Which IAM roles trust external accounts (not in AWS Organizations)? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-roles-trust-policies-allow | Do any IAM roles have trust policies that allow principals from other accounts without requiring MFA or an External ID? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-roles-trust-policies-allow-2 | Are there IAM roles whose trust policies allow any AWS principal (e.g., Principal set to "*")? | privilege-exposure | single-platform | 1 | easy | aws |
| aws-root-active-509-certificates | Does the root account have any active X.509 certificates? | auth-posture | single-platform | 1 | easy | aws |
| aws-root-active-access-keys | Does the root account have any active access keys? | auth-posture | single-platform | 1 | easy | aws |
| aws-root-mfa-enforced | Is root MFA enforced? | auth-posture | single-platform | 1 | easy | aws |
| aws-root-used-last-90 | Has the root account been used within the last 90 days? | dormant-stale | single-platform | 1 | easy | aws |
| aws-secrets-resource-owner-type | List secrets in AWS per resource and per owner type (root, IAM user, role). | inventory-visibility | single-platform | 1 | easy | aws |
| aws-service-identified-membership-service-user | Do any service users (identified by membership in a service-user group) have console passwords enabled? | credential-hygiene | single-platform | 1 | medium | aws |
| cross-access-but-don-corresponding | Which users have AWS SSO access but don't have a corresponding employee record in our BambooHR system? | cross-vendor-exploration | cross-platform | 3+ | medium | aws, bamboohr, okta |
| cross-according-employees-classified-contractors | According to BambooHR, do any employees classified as contractors, external, or freelancers have AWS SSO account? | cross-vendor-exploration | cross-platform | 3+ | medium | aws, bamboohr, okta |
| cross-accounts-active-employees-terminated | Are there any Google Workspace accounts still active for employees who have been terminated? | lifecycle-offboarding | cross-platform | 3+ | hard | bamboohr, gws, okta |
| cross-accounts-don-match-current | Are there any Okta accounts that don't match a current employee or contractor in BambooHR? | cross-vendor-exploration | cross-platform | 2 | medium | bamboohr, okta |
| cross-active-access-key-secret | List AWS IAM Users who have an active Access Key (Secret) older than 90 days but have never logged into Okta. | dormant-stale | cross-platform | 2 | medium | aws, okta |
| cross-active-accounts-disabled | Are there any active GitHub accounts for users who have been disabled in Azure AD? | lifecycle-offboarding | cross-platform | 2 | medium | azure, github |
| cross-active-but-synced-through | List all users who have an active account in Google Workspace but are synced through Okta. | cross-vendor-exploration | cross-platform | 2 | medium | gws, okta |
| cross-active-organization-members-two-factor | Which active Okta users who are GitHub organization members have two-factor authentication disabled on GitHub, broken down by their organization role (MEMBER vs ADMIN)? | auth-posture | cross-platform | 2 | medium | github, okta |
| cross-active-phishing-resistant-mfa-factor | Which active Okta users that also have an AWS IAM Identity Store identity have no phishing-resistant MFA factor enrolled? | auth-posture | cross-platform | 2 | hard | aws, okta |
| cross-active-provisioned-identity-store | Which active Okta users are provisioned in AWS Identity Store, hold ADMIN permission on at least one GitHub repository, and have a Google Workspace administrator role? | privilege-exposure | cross-platform | 3+ | hard | aws, github, gws, okta |
| cross-active-role-assignments-matching | Which active Google Workspace users have role assignments, and do they have a matching Okta identity (including Okta status)? | cross-vendor-exploration | cross-platform | 2 | hard | gws, okta |
| cross-admin-access-wasn-provisioned | Who has admin access in Mongo that wasn't provisioned through Okta? | cross-vendor-exploration | cross-platform | 2 | medium | mongo, okta |
| cross-admin-level-access-inactive-missing | Which AWS SSO users with admin-level access have an inactive or missing Okta account? | cross-vendor-exploration | cross-platform | 2 | hard | aws, okta |
| cross-administrators-also-hold-primitive | Which Google Workspace administrators also hold a primitive Owner or Editor role in GCP? | privilege-exposure | cross-platform | 2 | medium | gcp, gws |
| cross-admins-aren-listed | Who are the Google Workspace admins that aren't listed as admins in Okta? | cross-vendor-exploration | cross-platform | 2 | hard | gws, okta |
| cross-admins-aren-listed-2 | Who are the Azure AD admins that aren't listed as admins in Okta? | cross-vendor-exploration | cross-platform | 2 | hard | azure, okta |
| cross-anyone-without-mfa-access | Can anyone without MFA can access our production systems S3 Buckets or AWS Databases? | auth-posture | cross-platform | 2 | hard | aws, okta |
| cross-credentials-owning-cannot-traced | Are there any AWS IAM credentials whose owning IAM user cannot be traced back to an active BambooHR employee through Okta? | cross-vendor-exploration | cross-platform | 3+ | hard | aws, bamboohr, okta |
| cross-deactivated-but-active | Are there any users that are deactivated in Okta but still active in AWS? | lifecycle-offboarding | cross-platform | 2 | hard | aws, okta |
| cross-deprovisioned-admin-access | Are there deprovisioned users in Okta with admin access in GitHub? | lifecycle-offboarding | cross-platform | 2 | hard | github, okta |
| cross-disabled-active-accounts | Are there any disabled Okta users who still have active Google Workspace accounts? | lifecycle-offboarding | cross-platform | 2 | medium | gws, okta |
| cross-documents-owned-longer-working | Which former employees still own Google Workspace documents (private or shared drives) after their termination? | lifecycle-offboarding | cross-platform | 2 | hard | bamboohr, gws |
| cross-employees-active-admin-role | Which employees have active admin role assignments in Okta and what is their department in BambooHR? | inventory-visibility | cross-platform | 2 | hard | bamboohr, okta |
| cross-employees-marked-terminated-authored | Which employees marked as terminated in BambooHR have authored commits in GitHub after their recorded termination date? | lifecycle-offboarding | cross-platform | 2 | medium | bamboohr, github |
| cross-exist-both-entraid-but | Which users exist in both EntraID and GWS but don't have 2SV enforced in Google Workspace? | auth-posture | cross-platform | 2 | medium | azure, gws |
| cross-external-email-addresses-access | Are there any external email addresses with access to any shared GWS files that are not managed through Okta? | cross-vendor-exploration | cross-platform | 2 | medium | gws, okta |
| cross-find-without-mfa-identify | Which GitHub organization members without MFA enabled also have AWS admin-level permissions or MongoDB Atlas admin roles (ORG_OWNER, ORG_BILLING_ADMIN, GROUP_OWNER)? | cross-vendor-exploration | cross-platform | 3+ | hard | aws, github, mongo |
| cross-haven-logged-90-days | Which AWS SSO users haven't logged into Okta in 90 days but are still active in Okta? | dormant-stale | cross-platform | 2 | medium | aws, okta |
| cross-haven-used-90-days | Which users haven't used GWS for 90 days but still listed as active in Okta? | dormant-stale | cross-platform | 2 | medium | gws, okta |
| cross-high-privilege-human-identities-type | List high-privilege human identities by type (AWS IAM, Okta). | inventory-visibility | cross-platform | 2 | medium | aws, okta |
| cross-left-company-but-access | Show me users who left the company but still have GitHub access. | lifecycle-offboarding | cross-platform | 2 | medium | bamboohr, github |
| cross-marketing-finance-granted-write | Are any marketing or finance users granted write access to our code base? | privilege-exposure | cross-platform | 2 | hard | bamboohr, github |
| cross-most-publicly-accessible-files | Which Okta users have the most publicly accessible files in Google Workspace? | privilege-exposure | cross-platform | 2 | hard | gws, okta |
| cross-non-it-non-r-employees-super | Do any non-IT or non-R&D employees have super admin access to Google Workspace? | privilege-exposure | cross-platform | 2 | medium | bamboohr, gws |
| cross-offboarded-employees-active-accounts | Do any offboarded employees still have active AWS accounts? | lifecycle-offboarding | cross-platform | 2 | hard | aws, bamboohr |
| cross-org-members-matching-employee | Are there any GitHub org members with no matching BambooHR employee record? | cross-vendor-exploration | cross-platform | 2 | medium | bamboohr, github |
| cross-org-members-provisioned-through | Are there any GitHub org members not provisioned through Okta? | cross-vendor-exploration | cross-platform | 2 | medium | github, okta |
| cross-organization-members-corresponding-identity | Which GitHub organization members have no corresponding identity in Azure AD? | cross-vendor-exploration | cross-platform | 2 | medium | azure, github |
| cross-organization-members-hold-primitive | Which GitHub organization members hold GCP primitive roles (Owner or Editor), and what is their GitHub role? | privilege-exposure | cross-platform | 2 | medium | gcp, github |
| cross-organization-members-two-factor-authentication | Which GitHub organization members with two-factor authentication disabled also hold direct user-level IAM role bindings in GCP, and what privileged roles do they have? | privilege-exposure | cross-platform | 2 | medium | gcp, github |
| cross-organization-owners-but-corresponding | Which GitHub users are Organization Owners but do not have a corresponding active account in Okta? | cross-vendor-exploration | cross-platform | 2 | medium | github, okta |
| cross-permissions-bypass-mfa | Which Google Workspace users with GCP IAM permissions can bypass MFA? | auth-posture | cross-platform | 2 | medium | gcp, gws |
| cross-registered-only-weak-mfa | Which Azure AD users registered only with weak MFA methods (SMS or phone call) are also GitHub organization members? | auth-posture | cross-platform | 3+ | hard | azure, github, okta |
| cross-super-admins-periodic-review | List all Super Admins (for periodic review). | inventory-visibility | cross-platform | 2 | medium | gws, okta |
| cross-terminated-deactivated | Are there any MongoDB Atlas users who are terminated in BambooHR or deactivated in Okta? | lifecycle-offboarding | cross-platform | 3+ | hard | bamboohr, mongo, okta |
| cross-terminated-employees-access | Which terminated employees still have AWS SSO access? | lifecycle-offboarding | cross-platform | 3+ | hard | aws, bamboohr, okta |
| cross-terminated-employees-both-active | Are there any terminated employees who have both an active Google Workspace account and direct user-level IAM role bindings in our GCP projects? | lifecycle-offboarding | cross-platform | 2 | hard | bamboohr, gws |
| cross-terminated-employees-fully-offboarded | Which terminated employees have not been fully offboarded from all connected systems (Okta, Google Workspace, Azure AD, or AWS SSO)? | lifecycle-offboarding | cross-platform | 3+ | hard | aws, azure, bamboohr, gws, okta |
| cross-terminated-employees-own-publicly | Do any terminated employees own publicly accessible files in their Google Drive personal storage? | lifecycle-offboarding | cross-platform | 2 | hard | bamboohr, gws |
| cross-terminated-external-workers-non-osbench | Which terminated external workers (non-Osbench Labs employees) in BambooHR still have an active Okta account or GitHub organization membership? | lifecycle-offboarding | cross-platform | 3+ | hard | bamboohr, github, okta |
| cross-terminated-read-permissions-shared | Which terminated users still have read permissions to any shared GWS documents not owned by them? | lifecycle-offboarding | cross-platform | 2 | medium | bamboohr, gws |
| cross-terminated-write-permissions-shared | Which terminated users still have write permissions to any shared GWS documents not owned by them? | lifecycle-offboarding | cross-platform | 2 | medium | bamboohr, gws |
| gws-admin-missing-2-step-verification | Which Google Workspace admin users are missing 2-Step Verification? | auth-posture | single-platform | 1 | easy | gws |
| gws-domains-restrict-third-party-applications | Are all domains configured to restrict third-party applications and disallow less-secure apps? | auth-posture | single-platform | 1 | easy | gws |
| gws-domains-single-sign-on-ensuring | Are Google Workspace domains configured with Single Sign-On ensuring that post-SSO verification is enabled for the primary SSO profile? | auth-posture | single-platform | 1 | medium | gws |
| gws-labeled-sensitive-enrolled-advanced | Are all users labeled as sensitive enrolled in Google's Advanced Protection Program? | auth-posture | single-platform | 1 | easy | gws |
| gws-mfa-method-sms-voice | Are there any users whose MFA method is configured as SMS or voice call? | auth-posture | single-platform | 1 | easy | gws |
| gws-minimum-password-length-15 | Is the minimum password length configured to 15 characters or more? | credential-hygiene | single-platform | 1 | easy | gws |
| gws-minimum-password-length-least | Is the minimum password length at least 12 characters? | credential-hygiene | single-platform | 1 | easy | gws |
| gws-new-2-step-verification-enrollment | Is the new user 2-Step Verification enrollment period configured between 1 and 7 days for each organization unit? | auth-posture | single-platform | 1 | easy | gws |
| gws-number-super-admin-inclusive | Is the number of Google Workspace Super Admin users between 2 and 8 inclusive? | privilege-exposure | single-platform | 1 | medium | gws |
| gws-password-expiration-policies-disabled | Are password expiration policies disabled for all organizational units? | credential-hygiene | single-platform | 1 | easy | gws |
| gws-password-policy-enforced-next | Is the password policy configured to be enforced at the next sign-in for all organizational units? | credential-hygiene | single-platform | 1 | easy | gws |
| gws-password-reuse-disabled | Is password reuse disabled for all organizational units? | credential-hygiene | single-platform | 1 | easy | gws |
| gws-privileged-accounts-super-admins | Do all privileged accounts (Super Admins and other high-privilege roles) authenticate with Google credentials and use phishing-resistant MFA? | auth-posture | single-platform | 1 | easy | gws |
| gws-recovery-options-disabled-super | Are account recovery options disabled for all users who are not Super Admins? | auth-posture | single-platform | 1 | easy | gws |
| gws-self-service-recovery-options-disabled | Are self-service account recovery options disabled for users assigned the Super Admin role? | auth-posture | single-platform | 1 | hard | gws |
| okta-active-but-signed-90 | Which Okta users are active but have not signed in for over 90 days? | dormant-stale | single-platform | 1 | easy | okta |
| okta-active-longer-members-group | Are there any active users who are no longer members of any group assigned through an Application or IdP? | dormant-stale | single-platform | 1 | hard | okta |
| okta-active-signed-90-days | Which active Okta users have not signed in for more than 90 days? | dormant-stale | single-platform | 1 | easy | okta |
| okta-applications-default-access-rules | Which applications' default access rules do not require MFA? | auth-posture | single-platform | 1 | easy | okta |
| okta-applications-mfa-required-false | Which applications have mfa_required set to FALSE or NULL? | auth-posture | single-platform | 1 | easy | okta |
| okta-connections-lack-signed-encrypted | Which Okta connections lack signed and encrypted SAML assertions? | auth-posture | single-platform | 1 | easy | okta |
| okta-identity-providers-idps-lack | Do any identity providers (IdPs) lack signed or encrypted SAML assertions? | auth-posture | single-platform | 1 | easy | okta |
| okta-legacy-authentication-factors-sms | Are any legacy authentication factors (SMS, voice, email, OTP) enabled? | auth-posture | single-platform | 1 | easy | okta |
| okta-mfa-policies-missing-phishing-resistant | Are MFA policies missing phishing-resistant authenticators? | auth-posture | single-platform | 1 | medium | okta |
| okta-password-policy-enforce-minimum | Does the password policy enforce a minimum length of at least 12 characters and require uppercase letters, lowercase letters, numbers and symbols? | credential-hygiene | single-platform | 1 | easy | okta |
| okta-sign-on-policies-exceed-120-minute | Which sign-on policies exceed a 120-minute session lifetime? | auth-posture | single-platform | 1 | easy | okta |
| okta-sign-on-policies-reauthentication-interval | Which sign-on policies have a reauthentication interval greater than 480 minutes (8 hours) or have no reauthentication interval set? | auth-posture | single-platform | 1 | easy | okta |
| okta-sign-on-policies-where-maximum | Are there any sign-on policies where the maximum session lifetime (max_session_lifetime_minutes) exceeds 120 minutes (2 hours)? | auth-posture | single-platform | 1 | easy | okta |
| okta-sign-on-policies-where-require | Are there any sign-on policies where require_mfa_each_signin is FALSE or NULL? | auth-posture | single-platform | 1 | easy | okta |

## Links

- [Methodology (markdown)](https://open.security/methodology.md): the five-step evaluation that scores every run
- [Leaderboards (markdown)](https://open.security/leaderboards.md): public standings and what each signal measures
- [Site brief](https://open.security/llms.txt): what open.security is, how to run and submit
- [ISPM benchmark (HTML)](https://open.security/benchmarks/)
