CUSP exposes scientific forecasting limits and uncertainty in automated judging

Engage research development and principal investigators who use assistants to assess promising directions. Ask what decisions a forecast influences, which errors are costly and how claims are reviewed today. Offer a bounded evaluation of one planning workflow using public historical questions. The value hypothesis is better identification of unsupported confidence. Do not promise reliable discovery prediction or use this benchmark to justify automated funding decisions. Include domain-review cost and explain that a fluent rationale is not observed scientific progress.

Build an evaluation harness with dated input snapshots, explicit retrieval boundaries and retained model versions. Keep proposed mechanisms separate from forecasts of realization. Use blinded domain review alongside deterministic scoring, and test unavailable-information cases. Proposed proof should compare the assisted workflow with an expert-led baseline on a preregistered local corpus, report task-specific denominators and measure calibration and reviewer disagreement. Audit the judging system independently; restricted hosting alone cannot resolve invalid questions or inappropriate acceptance criteria.

A principal investigator should own scientific acceptance with research software support and independent domain reviewers. Agree the corpus, rubric, stopping rules and data permissions before adoption. Train staff to preserve uncertainty and log disagreements. Proposed acceptance criteria include a traceable source for every factual input, reproducible scoring, adjudication of every consequential disagreement and a documented human release decision. Track correction time alongside output quality. Risks include retrospective leakage, selection bias, overstated confidence and treating generated grades as authoritative.