From the AWS edition of November 10, 2026
Irregular's evaluation of GPT-5.6 Sol finds real zero-day discovery but low end-to-end reliability, in an unmitigated test setting
Irregular · Cybersecurity research · Not jurisdiction-specific
- Publisher
- Irregular
- Original publication
- 2026-06-26
- Source retrieved
- 2026-11-11
- Event date
- 2026-06-26
What happened
Irregular, working with OpenAI, evaluated GPT-5.6 Sol on three offensive-security suites. The model solved 19 of 197 FrontierCyber real-system challenges, 7 of 11 long-horizon CyScenarioBench scenarios (28% average success across attempts) and all 22 medium and hard Atomic challenges at least once. It found new zero-days, including in a widely used database server. Irregular says the model remains unreliable at end-to-end operations and struggles against hardened targets. Testing excluded deployment mitigations. Why now: Daybreak Blue (GPT-5.6 Sol) became callable on Bedrock's runtime endpoint on November 10.
Why it matters
Gives SLED security leaders evidence on the model behind Daybreak Blue that is separate from AWS or OpenAI marketing: real capability, but low success rates on real systems and limits in long tasks. Also shows patch windows can shrink when models can turn fixes into exploits within hours.
Evidence and measured results
Independent lab evaluation conducted in collaboration with OpenAI; offensive capability elicitation without deployment mitigations, so it measures capability, not deployed misuse risk or defensive productivity.
Limitations and uncertainty
Collaboration with the model developer; unmitigated configuration; benchmarks are Irregular's own. Not a test of defensive use in public-sector environments. Published June 2026; filed now for the November 10 Bedrock availability.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-11; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Security leaders cannot tell from vendor announcements how capable or reliable cyber models are.
- Stakeholders
- CISO
- Vulnerability management lead
- Security architect
- Discovery questions
- How are vulnerability findings validated today?
- What is current time from public fix to patch deployment?
- Which code would a pilot cover?
- Value hypothesis
- Models can surface real vulnerabilities but need analyst validation; productivity effect unmeasured.
- Potential engagement
- An evidence briefing paired with a scoped, measured pilot design.
- Unsupported claims
- Autonomous security operations
- Deployed misuse risk
- Defensive productivity gains
Pre-sales engineering
Role takeaway
- Fit
- Context for any SLED evaluation of Daybreak Blue.
- Architecture and integration
- Informs human-in-the-loop design for Bedrock security workflows.
- Prerequisites
- Seeded test environment
- Reproduction tooling
- Analyst time
- Constraints
- Unmitigated test configuration
- Developer collaboration
- Low success on real systems
- Security
- Assume shortened exploit windows; prioritize patch speed.
- Proof of value
- Validated findings per analyst hour
- False-positive rate on own code
Delivery
Role takeaway
- Work
- Design a pilot with reproduction and patch-speed metrics.
- Dependencies
- Analyst capacity
- Patching process
- Ownership
- Vulnerability management.
- Skills and adoption
- Analysts learn to validate AI-generated exploits.
- Governance checkpoints
- Reproduction review
- Patch-time monitoring
- Acceptance criteria
- Proposed: every finding carries reproduction status
- Proposed: patch time measured before and after
- Risks
- Over-trust
- Remediation backlog
- Results not transferable to mitigated model
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
No architecture content; results inform how much human validation a Bedrock-based vulnerability workflow needs.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports treating cyber models as dual-use with strong access controls, and as aids rather than autonomous operators.
Security and privacy
What data, permissions and controls need testing?
Reports a case where an exploit was built five hours after a fix was merged, which argues for faster patching of public-sector systems.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
No accessibility content.
Procurement
What should contracts, pricing and exit terms secure?
Irregular reports cost per success similar to or slightly above GPT-5.5; no Bedrock pricing.
Operating model
Which teams own the service once it runs?
Low end-to-end reliability implies human analysts remain responsible for validation and remediation.
Publication history
- 2026-11-10AWS · Issue 128 resources
Stable resource ID: irregular-gpt-5-6-sol-offensive-security-assessment-20260626