{"resourceId":"harvard-ai-coding-agents-review-bottleneck-jellyfish-20261009","versions":[{"version":"external-c0c443ecf1f8a16cbe700ca6a919be3317a4be514820acabf9e8ab999cad3bc8","resource":{"id":"harvard-ai-coding-agents-review-bottleneck-jellyfish-20261009","title":"Harvard study of 700+ firms: AI coding agents raise code volume 30% but not software output as review time rises 49%","organization":"Fiona Chen and James Stratton (Harvard University), reported by Ars Technica; data from Jellyfish engineering analytics","sector":"Software engineering productivity research (limited relevance to research software)","geography":"700+ software firms (data 2021–March 2026)","publishedAt":"Ars Technica, Kyle Orland, 9 October 2026","publicationDate":"2026-10-09","eventDate":null,"sourceName":"Ars Technica","sourceLabel":"Credible reporting on an academic study","sourceUrl":"https://arstechnica.com/ai/2026/10/ai-coding-agents-generate-more-code-but-not-more-software/","evidenceClass":"independent-reporting","outcomeClass":"mixed","topics":["developers-agents","knowledge-work","operating-model"],"finding":"Using Jellyfish analytics (300 million work events, 700,000+ employees, 700+ firms, 2021–March 2026) and a difference-in-differences design, the researchers report AI coding agents raised lines of code 30%, commits 20% and pull requests 23%, but Jira issue/epic resolution did not change significantly. Review time from PR submission to merge rose 49%, the share of PRs with changes requested nearly doubled, comments per PR rose 35%, and the share of workers doing reviews rose 14%. AI agents produced 23.3% of review comments and 10.8% of PRs; no significant employment change attributed to AI.","sledRelevance":"Limited, transferable relevance: research software engineering groups and research computing teams adopting coding agents should expect review capacity, not code generation, to bound throughput. The firms are commercial, not universities.","evidence":"Opened the Ars Technica article via HTTPS (curl, full text extracted and read). The underlying Harvard paper was not opened; figures are as reported by Ars.","architectureImplications":"Interpretation: CI, automated tests and review tooling become the throughput constraint when agents generate code.","governanceImplications":"Code-review policies and accountability for agent-written code need explicit definition, especially for research code underpinning publications.","securityPrivacyImplications":"Higher volumes of agent-generated code increase review load for security-relevant changes; not measured in the study.","caveats":"Secondary reporting; primary paper not inspected; commercial firms only; data cutoff March 2026 predates newer agents; Jellyfish customers may not be representative; measures output proxies, not code quality or scientific correctness.","streamIds":["research"],"roles":{"sales":"Sales framing: buying coding agents does not by itself deliver more finished software, according to this firm-level study. For research computing and research software groups, ask how code review, testing and merge throughput are measured today. A bounded workflow assessment of review capacity fits. The evidence cannot support claims of faster delivery, staff reductions or applicability to university research code without local measurement.","engineering":"Engineering framing: in this study agent adoption raised commits and pull requests while review time rose 49% and change requests nearly doubled. Fit for research software teams depends on CI coverage, automated testing and review tooling that can absorb more code. Prerequisites are baseline metrics for PR cycle time. Validate on local repositories before and after agent adoption, and note the paper itself was not inspected here.","delivery":"Delivery framing: an agent rollout in a research software group should be judged by merged, tested features, not lines of code. Delivery would set baseline metrics, define review rules for agent-written code, and add test automation. Ownership sits with the research software engineering lead. Propose acceptance criteria on issue resolution and review cycle time; risks include reviewer overload and unreviewed agent code entering research pipelines."},"roleDetails":{"sales":{"customerProblem":"Research software teams adopting coding agents may see more code but no more delivered, reviewed software.","stakeholders":["Research software engineering leads","Research computing directors","Lab PIs maintaining shared code"],"discoveryQuestions":["How are PR cycle time and issue resolution measured today?","Who reviews agent-generated code?","Is test automation sufficient to absorb more changes?"],"valueHypothesis":"Investing in review and test capacity alongside agents may convert extra code into delivered features; unmeasured in research settings.","potentialEngagement":"Two-week baseline and review-capacity assessment before or during a coding-agent pilot.","unsupportedClaims":["Faster software delivery from agents","Staff reductions","Direct applicability to university research code"]},"engineering":{"fit":"Research software groups with Git-based workflows considering coding agents.","architectureIntegration":"Repository analytics for PR cycle time, CI pipelines with automated tests, and review tooling (interpretation).","prerequisites":["Baseline repository metrics","CI and test coverage","Defined review policy"],"constraints":["Commercial-firm sample","Data before newer agents","Primary paper not inspected"],"security":"Require human review of agent-written changes touching data access, credentials or published analysis code.","proofOfValue":["Compare PR cycle time before and after pilot","Track issue resolution rate, not commits","Measure share of agent PRs needing changes"]},"delivery":{"work":"Establish baseline metrics, define review rules for agent code, expand test automation, and monitor reviewer load.","dependencies":["Repository analytics access","Agent licenses","Reviewer availability"],"ownership":"Research software engineering lead with research computing management.","skillsAdoption":"Train developers in reviewing agent output and writing tests that catch agent errors.","governanceCheckpoints":["Review policy approval","Mid-pilot metrics review","Decision gate before broader rollout"],"acceptanceCriteria":["Proposed: issue resolution rate does not fall during pilot","Proposed: PR review time increase under 20%","Proposed: 100% of agent PRs human-reviewed"],"risks":["Reviewer overload","Unreviewed code in research pipelines","Over-reading output metrics"]}},"retrievedAt":"2026-10-11T05:16:38-05:00","enrichedAt":"2026-10-11T05:16:49-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Reviewer workload grows; teams need review skills and capacity planning rather than fewer staff.","procurementImplications":"Agent licenses alone do not raise delivered output; budget for review and testing capacity.","operatingModelImplications":"Shifts effort from writing to reviewing code; research software teams should plan staffing accordingly.","sourceVerification":{"openedUrl":"https://arstechnica.com/ai/2026/10/ai-coding-agents-generate-more-code-but-not-more-software/","referenceExcerpt":"the code review process significantly increases in length, pull requests are more likely to require revisions, and reviewers leave more comments.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}