{"resourceId":"cispa-llm-peer-review-confidence-style-bias-20261026","versions":[{"version":"external-5d4a37988e9c5d5db360e41eecda6c7cefabdc17d749be635322cfa02669c6fa","resource":{"id":"cispa-llm-peer-review-confidence-style-bias-20261026","title":"CISPA-led preprint: LLM peer reviewers reward confident writing over identical science, flipping accept/reject decisions in up to 21% of papers","organization":"CISPA Helmholtz Center for Information Security, with DFKI, University of Technology Nuremberg and INRIA","sector":"Scholarly peer review and research evaluation","geography":"Germany and France (authors); applicable to conferences, journals and university research offices internationally","publishedAt":"26 October 2026 (arXiv v1 submission)","publicationDate":"2026-10-26","eventDate":null,"sourceName":"arXiv","sourceLabel":"Macharla Vasu et al., \"Style over Substance? Linguistic Bias in LLM-based Peer Review,\" arXiv:2610.35254, 26 Oct 2026","sourceUrl":"https://arxiv.org/abs/2610.35254","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement"],"finding":"The authors sampled 504 ICLR 2026 papers (12 accepted and 12 rejected from each of 21 areas), used Claude Opus 5 to rewrite each along two dimensions while preserving content (native versus non-native associated English; high versus low expressed confidence), validated rewrites with linguistic and content-integrity measures (similarity scores near 1), and had six LLM reviewers (GPT-5.5, Claude Sonnet 4.6, Gemini 3.5 Flash, Gemma3, Qwen3 VL, GLM4.7) review each version. All six preferred high-confidence rewrites, 1.3 to 7.0 times as often as low-confidence ones (Claude Sonnet 4.6 7.0x, Gemma3 6.0x). Claude Sonnet 4.6 accepted only the high-confidence version in 21.0% of papers versus 2.8% for the reverse. Native-associated rewrites were favoured more weakly (1.0x to 1.3x, in five of six reviewers). On 126 papers, native plus high-confidence versus non-native plus low-confidence reached ratios up to 9.1x. Reviewer score standard deviation on repeated runs was about 0.5 points for four models, 0.85 for GLM4.7 and 1.24 for Gemini 3.5 Flash. The authors recommend stress-testing LLM reviewers with content-preserving rewrites and calibrating raw scores before comparing models.","sledRelevance":"Directly relevant to research offices and pre-submission review services weighing AI reviewers: it is a controlled counterweight to the Harvard working paper in this stream on 27 October, showing LLM review scores can move with rhetoric alone, which affects fairness for researchers writing cautiously or in a second language.","evidence":"Read the arXiv abstract and the full HTML paper, including Tables 2, 3 and 5 and the main-results text. Main-results Table 4 values were taken from the narrative text. Appendices were skimmed only.","architectureImplications":"Any AI review pipeline should include a stress-test harness that rewrites inputs in content-preserving ways and checks score stability before deployment.","governanceImplications":"Shows that AI-generated review scores should not drive accept/reject or funding decisions without calibration and human judgment; supports disclosure and audit requirements.","securityPrivacyImplications":"Rhetorical framing can steer LLM reviewers, a form of manipulation that institutions should test for; uploading unpublished manuscripts to commercial models also needs approved terms.","caveats":"Preprint, not peer reviewed (submitted to ICLR 2027). Rewrites were generated by one model (Claude Opus 5) and the corpus is AI conference papers only. Effects on human reviewers were not measured. Some table values read from text rather than the full table.","streamIds":["research"],"roles":{"sales":"Lead with assurance testing rather than more AI review. A 26 October preprint found six LLM reviewers rewarded confident wording over identical science, with one model accepting only the confident version of 21% of papers. Research offices piloting AI review need a way to check fairness first. Offer a bounded bias and stability assessment. Do not claim AI review is unbiased or that these results describe human reviewers; this is an unreviewed preprint on AI conference papers.","engineering":"Frame fit around a stress-test harness for AI reviewers. The paper rewrites papers in content-preserving ways, validates the rewrites with similarity and linguistic metrics, then compares review scores across versions. An institution can reuse that design on its own prompts and models. Prerequisites are a rewriting model, a sample of de-identified manuscripts and approved model access. Validate on local disciplines, because the study used only ICLR papers.","delivery":"Treat delivery as adding a bias-testing gate to any AI review service. Work covers building rewrite sets, running reviewers repeatedly to measure variance, reporting score shifts and setting thresholds. Ownership sits with the research development or research integrity office. Governance checkpoints keep humans responsible for decisions. Acceptance criteria are proposals, such as score shifts under rewrites within measured run-to-run variance, not results from the paper."},"roleDetails":{"sales":{"customerProblem":"Research offices and review services want to use AI reviewers but cannot tell whether scores reflect scientific quality or writing style.","stakeholders":["Vice president or dean for research","Research integrity officer","Research development office","Faculty senate or equity committee"],"discoveryQuestions":["Are AI tools used in any internal review, award or pre-submission process today?","How are AI review outputs checked for bias?","Which researcher groups would be most affected by style bias?"],"valueHypothesis":"A bias and stability assessment may show whether an AI review setup is fair enough for advisory use before wider rollout.","potentialEngagement":"A short assessment applying content-preserving rewrites to a sample of local manuscripts and reporting score stability per model.","unsupportedClaims":["That any LLM reviewer is free of bias","That human reviewers show the same effects","That results generalize beyond AI conference papers","That AI review can replace human decisions"]},"engineering":{"fit":"Fits institutions running or evaluating AI review of manuscripts, proposals or internal competitions.","architectureIntegration":"Rewrite generator, content-integrity checks (embedding and TF-IDF similarity), repeated reviewer calls, score comparison and reporting.","prerequisites":["Approved model access for unpublished documents","De-identified sample manuscripts","Review prompt and rubric"],"constraints":["Preprint not peer reviewed","Single rewriting model","Corpus limited to ICLR papers"],"security":"Run tests only on services approved for confidential research documents and remove author identifiers.","proofOfValue":["Measure run-to-run variance per reviewer model","Compare score shifts under high and low confidence rewrites","Check decision flips at the local acceptance threshold"]},"delivery":{"work":"Build rewrite sets, run reviewers repeatedly, quantify bias and variance, and set go/no-go thresholds for advisory use.","dependencies":["Sample manuscripts with consent","Model access","Research integrity staff time"],"ownership":"Research integrity or research development office.","skillsAdoption":"Staff learn to read AI review scores as uncalibrated signals and to run periodic stress tests.","governanceCheckpoints":["Bias test before launch","Human decision retained on all outcomes","Re-test after model or prompt changes"],"acceptanceCriteria":["Score shifts under content-preserving rewrites below run-to-run variance (proposal)","No decision flips above an agreed rate on the test set","Test report published to stakeholders"],"risks":["Disadvantaging hedged or second-language writing","False assurance from limited tests","Model updates changing behaviour"]}},"retrievedAt":"2026-10-28T22:08:48-05:00","enrichedAt":"2026-10-28T22:08:51-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Researchers who hedge more or write in a second language could be disadvantaged by uncalibrated AI review, an equity issue for research development offices.","procurementImplications":"Buyers of AI review tools should ask vendors for stability and bias test results under content-preserving rewrites.","operatingModelImplications":"AI review can be positioned as advisory feedback with human reviewers retaining decisions, plus periodic bias testing by the service owner.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.35254","referenceExcerpt":"Our experiments expose a critical vulnerability in LLM-based peer review: reviewers systematically reward confident rhetoric independent of scientific content","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}