{"resourceId":"llm-replicability-robustness-political-social-science-20261104","versions":[{"version":"external-2c4ce84ba1df08c7421f0e5e403639f09bec2da1e6740df819715f867e0f42d1","resource":{"id":"llm-replicability-robustness-political-social-science-20261104","title":"Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models","organization":"Purdue University; University of Western Ontario","sector":"University research (political/social science methods; research administration and reproducibility)","geography":"United States and Canada (Purdue; Western Ontario); methods are framed for political and social science LLM research generally","publishedAt":"4 November 2026 (arXiv 2611.06639v1); forthcoming 2027 as Chapter 5 in LLMs in Political and Social Science: Principles and Applications of Transformer Models, Oxford University Press","publicationDate":"2026-11-04","eventDate":null,"sourceName":"arXiv","sourceLabel":"Timoneda, J. C. & Vallejo Vera, S., \"Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models,\" arXiv:2611.06639, 4 Nov 2026 (forthcoming OUP 2027)","sourceUrl":"https://arxiv.org/abs/2611.06639","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","operating-model"],"finding":"Joan C. Timoneda (Purdue Political Science) and Sebastián Vallejo Vera (University of Western Ontario Political Science) address reproducibility when LLM outputs are probabilistic and vary across models and versions, with effects on downstream statistical results. They distinguish replicability, reproducibility, and robustness, and propose a probabilistic standard: a replication succeeds when it recovers the same distribution of results rather than identical outputs. The chapter introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether errors align. Using three datasets, it shows that measurement choice can reverse the sign of downstream regression coefficients. Design-based corrections that use gold-labeled subsamples restore validity but cost estimate precision; a stacked ensemble of open-weight models is proposed to recover much of that lost precision while retaining validity. The chapter closes with disclosure standards covering code, prompts, codebooks, training data, and trained models. Included now because this 4 Nov methods chapter was not in prior editions and speaks directly to research-administration standards for LLM-annotated science, complementary to this week’s agent-reproduction evaluations.","sledRelevance":"Highly relevant to university research offices, research-integrity units, and social-science data services writing disclosure and replication policies for LLM-classified or LLM-annotated datasets. Shows that measurement choice is not a cosmetic NLP detail—it can reverse downstream inferential claims.","evidence":"arXiv preprint of a forthcoming Oxford University Press book chapter, with conceptual distinctions, prompt-script guidance, within-model variation tests, error-structure analysis on three datasets, and a proposed stacked open-weight ensemble correction. Not yet the printed 2027 volume.","architectureImplications":"Treat LLM annotation as a measurement pipeline: pin model versions, store standardized prompt scripts and codebooks, retain gold-labeled subsamples for design-based correction, and consider stacked open-weight ensembles when a single model’s labels would drive regressions.","governanceImplications":"Replication policies should require distributional replication (not bit-identical outputs), disclosure of prompts, codebooks, and model versions, and checks that conclusions are robust to measurement choice. Sign-reversing downstream coefficients are a research-integrity issue, not only a methods footnote.","securityPrivacyImplications":"Social-science corpora and gold labels may include sensitive political speech or personally identifiable text. Sharing prompts, codebooks, and trained models as the chapter urges must be balanced against data-use agreements; the chapter’s disclosure list includes training data and trained models, which campuses may need to redact or share under controlled access.","caveats":"Preprint of a forthcoming 2027 handbook chapter rather than a standalone peer-reviewed article; examples are political and social-text classification, not wet-lab or HPC workloads. The stacked-ensemble improvement is proposed and illustrated in the chapter’s correction setting, not a universal guarantee. 4 Nov date is just before the preferred window; selected as newly relevant and previously uncovered.","streamIds":["research"],"roles":{"sales":"Evaluation framing: Purdue and Western Ontario political scientists show that when LLMs label social-science text, measurement choice can reverse the sign of downstream regression coefficients. They argue replication should recover a distribution of results rather than identical outputs, and they propose gold-labeled corrections plus a stacked ensemble of open-weight models to restore validity without sacrificing as much precision, along with disclosure of code, prompts, codebooks, training data, and trained models. Use this with VPR, research integrity, and social-science data services writing LLM-research rules. Do not sell a guaranteed precision fix; the chapter is methodological guidance with three-dataset illustrations, forthcoming as an OUP chapter.","engineering":"Architecture framing: pin model versions; store a standardized prompt script and codebook with every annotation run; measure within-model variation for both encoders and decoders; and keep a gold-labeled subsample so design-based corrections can be applied before anyone interprets a regression. If single-model labels are unstable, a stacked ensemble of open-weight few-shot models is the chapter’s proposed way to recover precision lost in the correction step. Log error-structure so reviewers can see whether models fail on the same items.","delivery":"Delivery framing: add an LLM-measurement checklist to campus replication deposits—prompt script, model version, codebook, gold-label subsample, and robustness of downstream signs. Research-data services can own the deposit template; PIs own substantively whether a sign flip changes the claim. Acceptance is a documented distributional replication attempt and a report on whether conclusions survive alternative measurements, not identical chat transcripts."},"roleDetails":{"sales":{"customerProblem":"LLM-annotated social-science datasets can change downstream statistical conclusions, including reversing regression-coefficient signs, when model or measurement choices vary.","stakeholders":["Vice President for Research / research integrity","Social-science associate deans and PIs","Library research-data services","IRB or research-compliance staff where human-text data apply"],"discoveryQuestions":["Do LLM annotation projects pin model versions and deposit prompt scripts?","Have you tested whether alternative measurements flip downstream signs?","Is there a gold-labeled subsample for design-based correction?","What disclosure is required for prompts, codebooks, and trained models?"],"valueHypothesis":"Standardized prompts, gold-label corrections, and stacked open-weight ensembles may keep downstream estimates valid and more precise than single-model pipelines, if campuses adopt the chapter’s disclosure and robustness checks.","potentialEngagement":"A methods-and-policy workshop that turns the chapter’s disclosure list into a campus deposit template and reruns one local LLM-annotation paper through a sign-robustness check.","unsupportedClaims":["That stacked ensembles always restore precision in every discipline","Bit-identical replicability of decoder outputs","That this chapter replaces domain-specific statistical review"]},"engineering":{"fit":"Strong for text-classification and annotation pipelines feeding regressions; weaker for non-text laboratory instruments.","architectureIntegration":"Raw corpus to standardized prompt script to pinned encoder/decoder runs to error-structure logs to optional stacked open-weight ensemble to gold-label design-based correction to downstream model.","prerequisites":["Gold-labeled subsample","Stored prompts and codebooks","Pinned model versions","Compute for repeated runs and ensembles"],"constraints":["Illustrated on three social-science datasets, not a universal benchmark","Forthcoming book chapter / preprint","Open-weight ensemble assumes usable local or licensed models","Disclosure of training data may conflict with data-use agreements"],"security":"Protect sensitive political or personal text in gold labels and training shares; apply least-privilege access to deposited models.","proofOfValue":["Replication recovers an agreed distributional summary","Downstream coefficient signs tested under alternative measurements","Correction-step precision compared for single model vs stacked ensemble"]},"delivery":{"work":"Draft a deposit checklist; select a pilot annotation study; run measurement-robustness and correction comparisons; publish disclosure artifacts.","dependencies":["PI data-use permissions","Research-data repository","Statistical reviewer for downstream models"],"ownership":"PI owns substantive claims; research-data services own deposit standards; research integrity owns policy language.","skillsAdoption":"Students and staff learn to report distributions, prompt scripts, and sign-robustness rather than a single labeled spreadsheet.","governanceCheckpoints":["No public inference from unpinned model versions","Sign-flip check before press or policy claims","Disclosure package reviewed against data-use limits"],"acceptanceCriteria":["Proposed: prompt script, codebook, and model version deposited","Proposed: gold-label correction documented if used","Proposed: report states whether downstream signs are robust"],"risks":["False certainty from a single LLM labeling pass","Privacy leakage via deposited training data or models","Treating handbook guidance as if it were a completed multi-lab trial"]}},"retrievedAt":"2026-11-07T22:08:56-06:00","enrichedAt":"2026-11-07T22:14:26-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Standardized prompt scripts and disclosure checklists can make replication work more teachable for students, while gold-label and ensemble steps add specialist statistical labor.","procurementImplications":"Annotation-platform purchases should require export of prompts, model versions, codebook, and repeatable batch runs—not only a UI that produces a labeled file.","operatingModelImplications":"Pushes research administration toward distributional replication standards and shared deposit templates rather than one-off LLM labeling by individual labs.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2611.06639","referenceExcerpt":"Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}