From the Research Universities edition of September 22, 2026
Horizon audit: 29 confirmed broken tasks across 20 public agent datasets (5,241 scanned)
Horizon Analytics Labs · Independent AI evaluation / benchmark integrity research · United States (public Harbor-hub vendor datasets; global benchmark users)
- Publisher
- Horizon Analytics Labs
- Original publication
- September 21, 2026 (Horizon Research)
- Source retrieved
- 2026-09-23
What happened
Horizon Research (21 Sep 2026) audited public task datasets from major vendors in the Harbor hub. After scanning 20 datasets containing 5,241 tasks, closer inspection of 239, and one-by-one review of 34, it confirmed 29 broken tasks. Failures fit five categories—answer contamination, incomplete tests, trivially gameable grading, spec-to-test/capability mismatch, and oracle failure or flaky grading—with contamination, incomplete tests, and gameable grading accounting for 39 of 54 category placements and typically producing flattering noise that makes models look better. Worked examples include recovering a finished fix from Git history inside a SWE-bench Pro image and graders that encode contradictory expected values.
Why it matters
Why now: 21 Sep still-unarchived independent scrutiny directly serving the stream mandate to distinguish benchmark results from reproducible scientific progress, published since the last research edition.
Evidence and measured results
20 datasets; 5,241 tasks scanned; 239 closer; 34 manual; 29 confirmed broken; 5 failure categories; 39/54 placements in flattering modes; concrete SWE-bench Pro and Apex Agents examples.
Limitations and uncertainty
Not a prevalence estimate for all benchmarks; Harbor-hub agent/coding focus rather than university wet-lab science; auditor is building a vendor quality index.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-22; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Public agent benchmark scores can flatter models when tasks leak answers, under-test instructions, or accept shortcuts.
- Stakeholders
- Research computing / ML platform
- Procurement
- AI evaluation owners
- Faculty adopting agent tools
- Discovery questions
- Which Harbor/public datasets do we trust?
- Do reference solutions still pass?
- How are task versions pinned?
- What failure modes would flatter our shortlist?
- Value hypothesis
- Dataset hygiene audits (as in 29 confirmed broken tasks across 5,241 scanned) can prevent false confidence in agent benchmarks if findings drive pin/replace decisions—without claiming a full market prevalence rate.
- Potential engagement
- Bounded benchmark hygiene review of the campus's current evaluation task sets.
- Unsupported claims
- 29 broken tasks = market-wide rate
- All named vendors' datasets are uniformly broken
- Audit replaces scientific domain evaluation
Pre-sales engineering
Role takeaway
- Fit
- Instruction/environment/reference/grader consistency audits for public agent task datasets.
- Architecture and integration
- Pin Harbor task digests in evaluation CI; add oracle re-runs and contamination checks before leaderboard use.
- Prerequisites
- Pinned dataset versions
- Image/history inspection
- Grader reproducibility
- Finding disposition workflow
- Constraints
- Harbor-hub focus
- Not university wet-lab science benches
- First release of continuing audit
- Security
- Treat leaked answers in images as evaluation supply-chain integrity risk; control who can mutate graders.
- Proof of value
- Confirmed broken-task dossiers
- Oracle pass/fail logs
- Pinned replacements after fixes
Delivery
Role takeaway
- Work
- Sample current evaluation tasks, classify failure modes, pin or replace broken items, and document flattering risks.
- Dependencies
- ML platform
- Procurement/vendor contacts
- Research computing CI
- Ownership
- Evaluation owners maintain pinned sets; vendors remediate confirmed breaks.
- Skills and adoption
- Train staff to inspect task images, tests, and reference solutions.
- Governance checkpoints
- Version pinning
- Contamination refuse list
- Public claim review before citing scores
- Acceptance criteria
- Audit memo for in-use datasets
- Broken tasks removed or fixed
- Oracle checks in CI
- Risks
- Overgeneralizing from small confirmed set
- Ignoring non-Harbor science benches
- Vendor disputes delaying pins
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Evaluation harnesses must pin task digests, re-run oracles, and screen images for leaked answers before using public agent benchmarks in research-AI selection.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurement and research leadership should treat un-audited public leaderboard scores as insufficient evidence for campus AI tool adoption.
Security and privacy
What data, permissions and controls need testing?
Contaminated task environments are an evaluation integrity/supply-chain issue; control who can alter graders and images.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Without funded evaluation hygiene, smaller research groups may over-trust public scores; shared audit playbooks can lower the cost of safe adoption.
Procurement
What should contracts, pricing and exit terms secure?
Require pinned dataset versions, oracle-pass evidence, and contamination checks in AI evaluation RFPs—not raw benchmark claims.
Operating model
Which teams own the service once it runs?
Creates an ongoing evaluation-integrity function alongside model selection; vendors remediate while campuses pin and refuse broken tasks.
Publication history
- 2026-09-22Research Universities · Issue 174 resources
Stable resource ID: horizon-harbor-benchmark-audit-20260921