{"resourceId":"horizon-harbor-benchmark-audit-20260921","versions":[{"version":"external-71a3c0f10c6799519aa6f1b73610087d291f2468da219b988a09aa159dabb753","resource":{"id":"horizon-harbor-benchmark-audit-20260921","title":"Horizon audit: 29 confirmed broken tasks across 20 public agent datasets (5,241 scanned)","organization":"Horizon Analytics Labs","sector":"Independent AI evaluation / benchmark integrity research","geography":"United States (public Harbor-hub vendor datasets; global benchmark users)","publishedAt":"September 21, 2026 (Horizon Research)","publicationDate":"2026-09-21","eventDate":null,"sourceName":"Horizon Analytics Labs","sourceLabel":"Independent research article — public agent benchmark dataset audit","sourceUrl":"https://www.horizonanalyticslabs.com/research/public-benchmark-dataset-audit","evidenceClass":"independent-research","outcomeClass":"cautionary","topics":["developers-agents","governance-procurement","data-security","knowledge-work"],"finding":"Horizon Research (21 Sep 2026) audited public task datasets from major vendors in the Harbor hub. After scanning 20 datasets containing 5,241 tasks, closer inspection of 239, and one-by-one review of 34, it confirmed 29 broken tasks. Failures fit five categories—answer contamination, incomplete tests, trivially gameable grading, spec-to-test/capability mismatch, and oracle failure or flaky grading—with contamination, incomplete tests, and gameable grading accounting for 39 of 54 category placements and typically producing flattering noise that makes models look better. Worked examples include recovering a finished fix from Git history inside a SWE-bench Pro image and graders that encode contradictory expected values.","sledRelevance":"Why now: 21 Sep still-unarchived independent scrutiny directly serving the stream mandate to distinguish benchmark results from reproducible scientific progress, published since the last research edition.","evidence":"20 datasets; 5,241 tasks scanned; 239 closer; 34 manual; 29 confirmed broken; 5 failure categories; 39/54 placements in flattering modes; concrete SWE-bench Pro and Apex Agents examples.","architectureImplications":"Evaluation harnesses must pin task digests, re-run oracles, and screen images for leaked answers before using public agent benchmarks in research-AI selection.","governanceImplications":"Procurement and research leadership should treat un-audited public leaderboard scores as insufficient evidence for campus AI tool adoption.","securityPrivacyImplications":"Contaminated task environments are an evaluation integrity/supply-chain issue; control who can alter graders and images.","caveats":"Not a prevalence estimate for all benchmarks; Harbor-hub agent/coding focus rather than university wet-lab science; auditor is building a vendor quality index.","streamIds":["research"],"roles":{"sales":"Sales framing: Buyers using public agent benchmarks to pick research/coding AI need hygiene checks—not headline scores. Horizon's 21 Sep audit supports a cautionary conversation: across 20 Harbor-hub datasets and 5,241 tasks it confirmed 29 broken tasks where contamination, incomplete tests, or gameable graders often make models look better. Discovery should ask which datasets a campus relies on, whether reference solutions pass, and how vendor tasks are pinned. Unsupported: that 29 broken tasks estimate market-wide prevalence, that all Scale AI/Mercor/OpenThoughts tasks fail, or that this alone proves any university AI scientist stack is unsafe.","engineering":"Engineering framing: Fit is dataset-level audit of instruction–environment–reference–grader agreement for agent/coding benchmarks used in evaluation harnesses. Integration points are Harbor-style task images, graders, and CI that re-runs published oracles—not swapping the campus LLM. Prerequisites include pinned task versions, ability to inspect images/history, and reproducible grader runs. Constraints: Harbor-hub vendor sample; confirmed count is not prevalence; auditor builds a vendor index. Security must treat contaminated images as supply-chain risk for evaluation integrity. Proof of value is confirmed broken-task dossiers with verifier evidence—not aggregate marketing scores.","delivery":"Delivery framing: Before adopting a public agent benchmark for research-AI evaluation, run a bounded hygiene pass: sample tasks, check for answer contamination, ensure tests cover the instruction, and verify the published solution still passes. Dependencies: ML platform team, procurement, and research computing. Ownership: evaluation owners pin datasets and record audit findings; vendors own fixes. Skills adoption trains staff to inspect task images and graders. Governance checkpoints: pin versions, refuse contaminated tasks, and document flattering-failure modes. Acceptance: written audit sample with disposition; risks include overgeneralizing from 29 tasks and ignoring university-science benchmarks not in Harbor."},"roleDetails":{"sales":{"customerProblem":"Public agent benchmark scores can flatter models when tasks leak answers, under-test instructions, or accept shortcuts.","stakeholders":["Research computing / ML platform","Procurement","AI evaluation owners","Faculty adopting agent tools"],"discoveryQuestions":["Which Harbor/public datasets do we trust?","Do reference solutions still pass?","How are task versions pinned?","What failure modes would flatter our shortlist?"],"valueHypothesis":"Dataset hygiene audits (as in 29 confirmed broken tasks across 5,241 scanned) can prevent false confidence in agent benchmarks if findings drive pin/replace decisions—without claiming a full market prevalence rate.","potentialEngagement":"Bounded benchmark hygiene review of the campus's current evaluation task sets.","unsupportedClaims":["29 broken tasks = market-wide rate","All named vendors' datasets are uniformly broken","Audit replaces scientific domain evaluation"]},"engineering":{"fit":"Instruction/environment/reference/grader consistency audits for public agent task datasets.","architectureIntegration":"Pin Harbor task digests in evaluation CI; add oracle re-runs and contamination checks before leaderboard use.","prerequisites":["Pinned dataset versions","Image/history inspection","Grader reproducibility","Finding disposition workflow"],"constraints":["Harbor-hub focus","Not university wet-lab science benches","First release of continuing audit"],"security":"Treat leaked answers in images as evaluation supply-chain integrity risk; control who can mutate graders.","proofOfValue":["Confirmed broken-task dossiers","Oracle pass/fail logs","Pinned replacements after fixes"]},"delivery":{"work":"Sample current evaluation tasks, classify failure modes, pin or replace broken items, and document flattering risks.","dependencies":["ML platform","Procurement/vendor contacts","Research computing CI"],"ownership":"Evaluation owners maintain pinned sets; vendors remediate confirmed breaks.","skillsAdoption":"Train staff to inspect task images, tests, and reference solutions.","governanceCheckpoints":["Version pinning","Contamination refuse list","Public claim review before citing scores"],"acceptanceCriteria":["Audit memo for in-use datasets","Broken tasks removed or fixed","Oracle checks in CI"],"risks":["Overgeneralizing from small confirmed set","Ignoring non-Harbor science benches","Vendor disputes delaying pins"]}},"retrievedAt":"2026-09-22T22:17:16-05:00","enrichedAt":"2026-09-22T22:17:16-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Without funded evaluation hygiene, smaller research groups may over-trust public scores; shared audit playbooks can lower the cost of safe adoption.","procurementImplications":"Require pinned dataset versions, oracle-pass evidence, and contamination checks in AI evaluation RFPs—not raw benchmark claims.","operatingModelImplications":"Creates an ongoing evaluation-integrity function alongside model selection; vendors remediate while campuses pin and refuse broken tasks.","sourceVerification":{"openedUrl":"https://www.horizonanalyticslabs.com/research/public-benchmark-dataset-audit","referenceExcerpt":"Across 20 datasets and 5,241 tasks, we confirmed 29 broken tasks","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}