{"resourceId":"labfactory-executable-ai-labs-20260923","versions":[{"version":"external-0eda27531243145ce9fcf56c3afec4cdb37a9dbe84ec2e22deb1ad0972864a06","resource":{"id":"labfactory-executable-ai-labs-20260923","title":"LabFactory: AI builder delivers executable AI labs; 28 selected constructions pass all 33 host-side subtests","organization":"University of Oxford / UCL / Technical University of Munich / Oxford-Suzhou Centre (author affiliations on arXiv)","sector":"Academic AI-for-science research","geography":"International (UK/EU/China affiliations; arXiv preprint)","publishedAt":"September 23, 2026 (arXiv submitted)","publicationDate":"2026-09-23","eventDate":null,"sourceName":"arXiv:2609.28697","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2609.28697","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","knowledge-work","governance-procurement","operating-model"],"finding":"Wu et al. (arXiv 2609.28697, submitted 23 Sep 2026) present LabFactory: an AI builder turns a scientific brief into an executable AI lab (task-specific solver with models, tools, knowledge, controller behind a fixed interface) in a metered workspace; a separate host executes the artifact on held-out inputs with reference labels outside the solver interface. They document 28 selected constructions across seven categories (molecular/genomic prediction through clinical decision support and biomedical text) whose delivered labs exceeded configured reference values on all 33 subtests under host-side execution; ten include models fitted during construction.","sledRelevance":"Research computing and research administration should evaluate AI-for-science builders by host-side held-out execution of delivered labs—not by the builder agent’s self-report—aligning with reward-hacking defenses that keep labels outside agent control.","evidence":"Opened arXiv abstract and HTML; 28 constructions, seven categories, all 33 subtests, host-side execution, and label-outside-interface design confirmed in abstract/HTML.","architectureImplications":"Split builder workspace from evaluation host; package solvers behind a fixed interface; keep reference labels off the solver input path.","governanceImplications":"Selected successful cases are not a random field sample; require disclosure of failures/non-selected builds before treating as procurement evidence.","securityPrivacyImplications":"Clinical/decision-support and biomedical text labs imply PHI/data-use controls on both builder workspace and host eval; paper is methodological, not a compliance certification.","caveats":"Preprint; constructions are selected successes; not a guarantee for arbitrary briefs; clinical categories need local regulatory review; resource limits are study-configured.","streamIds":["research"],"roles":{"sales":"Sales framing: Research leaders want AI that delivers a working, re-runnable lab—not a chat transcript. LabFactory’s 23 Sep preprint shows a builder/host split where delivered labs, not builder narratives, are scored: 28 selected constructions across seven scientific categories cleared all 33 host-side subtests with labels outside the solver interface. Use it to sell evaluation architecture (metered build + sealed host), while stressing these are selected cases—not universal success rates.","engineering":"Engineering framing: Implement builder sandboxes with metering, a fixed delivery interface, and a separate host that runs held-out evals without exposing reference labels to the solver. Support both fitted predictive models and tool/retrieval assemblies around a platform LLM. Instrument construction records so labs remain invocable and inspectable after the builder stops, matching integrity needs for sealed evaluation.","delivery":"Delivery framing: Pilot LabFactory-style builds on two to three non-clinical tasks first: freeze briefs and reference metrics, run host-side scoring with labels sealed from the solver, and archive delivered artifacts for later invocation. Acceptance is host-side clearance on held-out inputs plus inspectability—not builder self-scores. Gate clinical categories behind data-governance approval and require failure disclosure alongside showcase constructions."},"roleDetails":{"sales":{"customerProblem":"AI-for-science tools often report progress without leaving a re-executable system that independent hosts can score.","stakeholders":["Research computing","PIs in computational biology/medicine","Research integrity","Data governance","AI platform team"],"discoveryQuestions":["Who runs held-out evaluation?","Are labels visible to the builder?","What is the delivery interface?","How are failed builds reported?"],"valueHypothesis":"Builder/host separation with sealed labels can produce executable labs that clear reference metrics under independent execution for selected tasks.","potentialEngagement":"Pilot two scientific briefs through a metered builder with host-side scoring and artifact handoff.","unsupportedClaims":["All briefs succeed","Clinical deployment readiness","Random-sample success rate of 28/28","Replacement of human scientific judgment"]},"engineering":{"fit":"Metered AI builder workspace + separate evaluation host + fixed solver interface for executable AI labs.","architectureIntegration":"Wire object storage for delivered labs, sealed label stores, scoring jobs, and audit of construction traces; isolate builder network from label host.","prerequisites":["Task briefs with reference metrics","Held-out datasets","Metering/quotas","Containerized delivery format"],"constraints":["Selected constructions only","Seven study categories","Platform LLM dependency for many labs","Preprint status"],"security":"Keep reference labels and PHI off builder inputs; control egress; review clinical text/decision-support data paths.","proofOfValue":["Host-side subtest clearance","Post-construction invocability","Label non-exposure test","Comparison vs raw platform LLM where reported"]},"delivery":{"work":"Define briefs/metrics, run builder pilots, score on host, archive labs and failures.","dependencies":["Data stewards","Compute quotas","PI reviewers","Integrity/governance for clinical scopes"],"ownership":"Platform owns builder/host split; PI owns brief and scientific acceptance; governance owns clinical data use.","skillsAdoption":"Teach PIs to write briefs with measurable reference values and to inspect delivered solvers.","governanceCheckpoints":["Brief/metric freeze","Data-class approval","Host-side result review","Failure disclosure"],"acceptanceCriteria":["Host clears agreed subtests","Labels confirmed outside solver inputs","Artifact rerun documented"],"risks":["Selection bias if only successes published","Clinical overreach","Builder contaminates eval","Metric gaming without sealed labels"]}},"retrievedAt":"2026-09-26T05:20:58-05:00","enrichedAt":"2026-09-26T05:21:25-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Executable labs can lower recurring coding burden for domain scientists if briefs and host evals are supported; still needs platform skills for metering and scoring.","procurementImplications":"Buy or build AI-for-science systems that deliver host-scored artifacts with sealed labels; require failure reporting, not only showcase constructions.","operatingModelImplications":"Operating model separates construction (metered builder) from acceptance (independent host), matching integrity needs from reward-hacking research.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.28697","referenceExcerpt":"delivered labs exceeded their configured reference values on all 33 subtests under host-side execution.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}