{"resourceId":"irregular-gpt-5-6-sol-offensive-security-assessment-20260626","versions":[{"version":"external-68342420a81c71ffacec2dc3f16dca75a5abb78c99a08b31a6ab3dc78c9986fb","resource":{"id":"irregular-gpt-5-6-sol-offensive-security-assessment-20260626","title":"Irregular's evaluation of GPT-5.6 Sol finds real zero-day discovery but low end-to-end reliability, in an unmitigated test setting","organization":"Irregular","sector":"Cybersecurity research","geography":"Not jurisdiction-specific","publishedAt":"2026-06-26","publicationDate":"2026-06-26","eventDate":"2026-06-26","sourceName":"Irregular","sourceLabel":"Assessing GPT-5.6 Sol Against Offensive Security Benchmarks","sourceUrl":"https://www.irregular.com/research/assessing-gpt-5.6-sol","evidenceClass":"independent-research","outcomeClass":"mixed","topics":["data-security","developers-agents"],"finding":"Irregular, working with OpenAI, evaluated GPT-5.6 Sol on three offensive-security suites. The model solved 19 of 197 FrontierCyber real-system challenges, 7 of 11 long-horizon CyScenarioBench scenarios (28% average success across attempts) and all 22 medium and hard Atomic challenges at least once. It found new zero-days, including in a widely used database server. Irregular says the model remains unreliable at end-to-end operations and struggles against hardened targets. Testing excluded deployment mitigations. Why now: Daybreak Blue (GPT-5.6 Sol) became callable on Bedrock's runtime endpoint on November 10.","sledRelevance":"Gives SLED security leaders evidence on the model behind Daybreak Blue that is separate from AWS or OpenAI marketing: real capability, but low success rates on real systems and limits in long tasks. Also shows patch windows can shrink when models can turn fixes into exploits within hours.","evidence":"Independent lab evaluation conducted in collaboration with OpenAI; offensive capability elicitation without deployment mitigations, so it measures capability, not deployed misuse risk or defensive productivity.","architectureImplications":"No architecture content; results inform how much human validation a Bedrock-based vulnerability workflow needs.","governanceImplications":"Supports treating cyber models as dual-use with strong access controls, and as aids rather than autonomous operators.","securityPrivacyImplications":"Reports a case where an exploit was built five hours after a fix was merged, which argues for faster patching of public-sector systems.","caveats":"Collaboration with the model developer; unmitigated configuration; benchmarks are Irregular's own. Not a test of defensive use in public-sector environments. Published June 2026; filed now for the November 10 Bedrock availability.","streamIds":["aws"],"roles":{"sales":"Interpretation: use this to set realistic expectations when a CISO asks about Daybreak on Bedrock. The customer problem is judging whether cyber models deliver value worth their access burden. Stakeholders are the CISO and vulnerability management lead. Discovery should ask how findings are validated today. Present the evidence as real but unreliable capability that needs analysts; avoid claims of autonomous security or reduced headcount.","engineering":"Interpretation: engineers should design any Daybreak workflow with human validation as a required step. The evaluation shows low success on real targets and weakness on long tasks, so pipelines should break work into scoped tasks with checkpoints. Prerequisites are a seeded test environment and reproduction tooling. Constraint: results came from an unmitigated configuration, so deployed behavior may differ. Proof of value should measure validated findings per analyst hour on the customer's own code.","delivery":"Interpretation: delivery should embed reproduction and patch-speed metrics. Dependencies are analyst capacity and a patching process. Ownership sits with vulnerability management. Governance checkpoints include reviewing reproduced findings and monitoring time from fix to deployment. Proposed acceptance criteria: findings tracked with reproduction status, and patch time measured before and after the pilot. Risks include over-trust in model findings and backlog growth without remediation capacity."},"roleDetails":{"sales":{"customerProblem":"Security leaders cannot tell from vendor announcements how capable or reliable cyber models are.","stakeholders":["CISO","Vulnerability management lead","Security architect"],"discoveryQuestions":["How are vulnerability findings validated today?","What is current time from public fix to patch deployment?","Which code would a pilot cover?"],"valueHypothesis":"Models can surface real vulnerabilities but need analyst validation; productivity effect unmeasured.","potentialEngagement":"An evidence briefing paired with a scoped, measured pilot design.","unsupportedClaims":["Autonomous security operations","Deployed misuse risk","Defensive productivity gains"]},"engineering":{"fit":"Context for any SLED evaluation of Daybreak Blue.","architectureIntegration":"Informs human-in-the-loop design for Bedrock security workflows.","prerequisites":["Seeded test environment","Reproduction tooling","Analyst time"],"constraints":["Unmitigated test configuration","Developer collaboration","Low success on real systems"],"security":"Assume shortened exploit windows; prioritize patch speed.","proofOfValue":["Validated findings per analyst hour","False-positive rate on own code"]},"delivery":{"work":"Design a pilot with reproduction and patch-speed metrics.","dependencies":["Analyst capacity","Patching process"],"ownership":"Vulnerability management.","skillsAdoption":"Analysts learn to validate AI-generated exploits.","governanceCheckpoints":["Reproduction review","Patch-time monitoring"],"acceptanceCriteria":["Proposed: every finding carries reproduction status","Proposed: patch time measured before and after"],"risks":["Over-trust","Remediation backlog","Results not transferable to mitigated model"]}},"retrievedAt":"2026-11-11T04:01:04Z","enrichedAt":"2026-11-11T04:05:52Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"No accessibility content.","procurementImplications":"Irregular reports cost per success similar to or slightly above GPT-5.5; no Bedrock pricing.","operatingModelImplications":"Low end-to-end reliability implies human analysts remain responsible for validation and remediation.","sourceVerification":{"openedUrl":"https://www.irregular.com/research/assessing-gpt-5.6-sol","referenceExcerpt":"the model remains unreliable at autonomously carrying out many types of cyber operations end to end.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}