From the Research Universities edition of October 28, 2026
CISPA-led preprint: LLM peer reviewers reward confident writing over identical science, flipping accept/reject decisions in up to 21% of papers
CISPA Helmholtz Center for Information Security, with DFKI, University of Technology Nuremberg and INRIA · Scholarly peer review and research evaluation · Germany and France (authors); applicable to conferences, journals and university research offices internationally
- Publisher
- arXiv
- Original publication
- 26 October 2026 (arXiv v1 submission)
- Source retrieved
- 2026-10-29
What happened
The authors sampled 504 ICLR 2026 papers (12 accepted and 12 rejected from each of 21 areas), used Claude Opus 5 to rewrite each along two dimensions while preserving content (native versus non-native associated English; high versus low expressed confidence), validated rewrites with linguistic and content-integrity measures (similarity scores near 1), and had six LLM reviewers (GPT-5.5, Claude Sonnet 4.6, Gemini 3.5 Flash, Gemma3, Qwen3 VL, GLM4.7) review each version. All six preferred high-confidence rewrites, 1.3 to 7.0 times as often as low-confidence ones (Claude Sonnet 4.6 7.0x, Gemma3 6.0x). Claude Sonnet 4.6 accepted only the high-confidence version in 21.0% of papers versus 2.8% for the reverse. Native-associated rewrites were favoured more weakly (1.0x to 1.3x, in five of six reviewers). On 126 papers, native plus high-confidence versus non-native plus low-confidence reached ratios up to 9.1x. Reviewer score standard deviation on repeated runs was about 0.5 points for four models, 0.85 for GLM4.7 and 1.24 for Gemini 3.5 Flash. The authors recommend stress-testing LLM reviewers with content-preserving rewrites and calibrating raw scores before comparing models.
Why it matters
Directly relevant to research offices and pre-submission review services weighing AI reviewers: it is a controlled counterweight to the Harvard working paper in this stream on 27 October, showing LLM review scores can move with rhetoric alone, which affects fairness for researchers writing cautiously or in a second language.
Evidence and measured results
Read the arXiv abstract and the full HTML paper, including Tables 2, 3 and 5 and the main-results text. Main-results Table 4 values were taken from the narrative text. Appendices were skimmed only.
Limitations and uncertainty
Preprint, not peer reviewed (submitted to ICLR 2027). Rewrites were generated by one model (Claude Opus 5) and the corpus is AI conference papers only. Effects on human reviewers were not measured. Some table values read from text rather than the full table.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-28; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research offices and review services want to use AI reviewers but cannot tell whether scores reflect scientific quality or writing style.
- Stakeholders
- Vice president or dean for research
- Research integrity officer
- Research development office
- Faculty senate or equity committee
- Discovery questions
- Are AI tools used in any internal review, award or pre-submission process today?
- How are AI review outputs checked for bias?
- Which researcher groups would be most affected by style bias?
- Value hypothesis
- A bias and stability assessment may show whether an AI review setup is fair enough for advisory use before wider rollout.
- Potential engagement
- A short assessment applying content-preserving rewrites to a sample of local manuscripts and reporting score stability per model.
- Unsupported claims
- That any LLM reviewer is free of bias
- That human reviewers show the same effects
- That results generalize beyond AI conference papers
- That AI review can replace human decisions
Pre-sales engineering
Role takeaway
- Fit
- Fits institutions running or evaluating AI review of manuscripts, proposals or internal competitions.
- Architecture and integration
- Rewrite generator, content-integrity checks (embedding and TF-IDF similarity), repeated reviewer calls, score comparison and reporting.
- Prerequisites
- Approved model access for unpublished documents
- De-identified sample manuscripts
- Review prompt and rubric
- Constraints
- Preprint not peer reviewed
- Single rewriting model
- Corpus limited to ICLR papers
- Security
- Run tests only on services approved for confidential research documents and remove author identifiers.
- Proof of value
- Measure run-to-run variance per reviewer model
- Compare score shifts under high and low confidence rewrites
- Check decision flips at the local acceptance threshold
Delivery
Role takeaway
- Work
- Build rewrite sets, run reviewers repeatedly, quantify bias and variance, and set go/no-go thresholds for advisory use.
- Dependencies
- Sample manuscripts with consent
- Model access
- Research integrity staff time
- Ownership
- Research integrity or research development office.
- Skills and adoption
- Staff learn to read AI review scores as uncalibrated signals and to run periodic stress tests.
- Governance checkpoints
- Bias test before launch
- Human decision retained on all outcomes
- Re-test after model or prompt changes
- Acceptance criteria
- Score shifts under content-preserving rewrites below run-to-run variance (proposal)
- No decision flips above an agreed rate on the test set
- Test report published to stakeholders
- Risks
- Disadvantaging hedged or second-language writing
- False assurance from limited tests
- Model updates changing behaviour
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Any AI review pipeline should include a stress-test harness that rewrites inputs in content-preserving ways and checks score stability before deployment.
Governance
Who approves, reviews and stays accountable for outcomes?
Shows that AI-generated review scores should not drive accept/reject or funding decisions without calibration and human judgment; supports disclosure and audit requirements.
Security and privacy
What data, permissions and controls need testing?
Rhetorical framing can steer LLM reviewers, a form of manipulation that institutions should test for; uploading unpublished manuscripts to commercial models also needs approved terms.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Researchers who hedge more or write in a second language could be disadvantaged by uncalibrated AI review, an equity issue for research development offices.
Procurement
What should contracts, pricing and exit terms secure?
Buyers of AI review tools should ask vendors for stability and bias test results under content-preserving rewrites.
Operating model
Which teams own the service once it runs?
AI review can be positioned as advisory feedback with human reviewers retaining decisions, plus periodic bias testing by the service owner.
Publication history
- 2026-10-28Research Universities · Issue 505 resources
Stable resource ID: cispa-llm-peer-review-confidence-style-bias-20261026