From the Student Success edition of October 1, 2026
Fairness Theatre: post-hoc fairness on vendor-controlled college EWS redistributes disparities without fixing them
University of Toronto & Georgia Institute of Technology (McConvey, Zhai, Li, Guha) · Higher education advising / early warning systems / algorithmic fairness · Canada (public college in Ontario; research evaluation under simulated procurement constraints)
- Publisher
- arXiv
- Original publication
- Submitted to arXiv 29 September 2026 (cs.HC)
- Source retrieved
- 2026-10-02
What happened
Using records from a public college in Ontario, the authors evaluate six post-hoc fairness interventions on a research replica of a proprietary Early Warning System under simulated procurement constraints that leave institutions able to adjust outputs but not model design. Interventions redistributed disparities without consistently reducing them; some approaches favored already-advantaged groups when disadvantage was defined via group size; small groups remained poorly served. They term the resulting condition fairness theatre: institutions perform fairness work while groups’ error burdens persist or worsen because vendors retain design control.
Why it matters
Direct HE advising/retention systems signal: equity overlays on vendor-locked early-warning products can look like fairness work without improving who is wrongly flagged or missed. Complements tutoring equity findings by showing procurement structure—not only model choice—limits student-success fairness.
Evidence and measured results
Empirical evaluation of six post-hoc methods on college EWS replica with error-type profiling. Simulated constraints; single college; preprint; fairness metrics do not establish retention gains.
Limitations and uncertainty
Single Ontario college; simulated procurement; research replica not live vendor A/B; fairness metrics do not establish that interventions improve retention; preprint.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-02; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Colleges buy vendor EWS tools and try to fix inequity with post-hoc fairness settings they do not control at the model layer.
- Stakeholders
- student success
- institutional research
- advising leadership
- procurement
- equity/compliance
- Discovery questions
- Can we change features, labels, and objectives—or only thresholds?
- What are false-positive/negative rates by group today?
- What happens to small groups under each fairness method?
- Value hypothesis
- Procurement that requires design transparency and group-wise error accountability prevents fairness theatre from substituting for equitable advising support.
- Potential engagement
- EWS fairness validation and contract redesign using local records and error-type profiling.
- Unsupported claims
- Post-hoc fairness guarantees equity
- Risk scores equal need for help
- Retention lift from fairness overlays alone
Pre-sales engineering
Role takeaway
- Fit
- Vendor or in-house early-warning systems used for college advising outreach under procurement constraints.
- Architecture and integration
- Score service → group-wise error monitor → optional post-hoc adjuster with audit log → advisor UI with override; prefer redesign access over post-processing-only.
- Prerequisites
- historical outcomes labels
- group attributes under privacy policy
- advisor override channel
- Constraints
- vendor-locked model design
- small-group instability
- single-site evidence
- Security
- Sensitive demographic/proxy features; minimize; audit who sees risk scores; no automated high-stakes penalties from opaque scores.
- Proof of value
- Reduced harmful FN/FP for underserved groups
- Advisor override usability
- Documented ability to change model design or exit
Delivery
Role takeaway
- Work
- Profile baseline errors, test fairness overlays offline, brief advising leads, update procurement language, monitor live error burdens.
- Dependencies
- IR/data warehouse
- vendor cooperation
- advising managers
- legal/procurement
- Ownership
- Student-success analytics lead with procurement co-ownership of vendor terms.
- Skills and adoption
- Advisors interpret group error reports and override; IR maintains fairness monitoring.
- Governance checkpoints
- pre-overlay error baseline
- procurement design-access review
- quarterly disparity/error audit
- Acceptance criteria
- Group-wise FN/FP published internally
- Override path live
- Contract addresses design opacity
- Risks
- Fairness theatre
- Small-group neglect
- Stigmatizing flags
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Prefer systems where institutions can inspect features, labels, and training objectives—not only threshold/post-processing knobs. Log false-positive/negative burdens by group before and after any fairness overlay.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurement should require model-design transparency and contestability, not only post-hoc fairness dashboards. Treat fairness theatre as a contract failure mode when vendors lock design.
Security and privacy
What data, permissions and controls need testing?
EWS features may include sensitive proxies; minimize, document, and avoid stigmatizing outreach from opaque scores.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
When small or disadvantaged groups stay poorly served after fair overlays, advisors inherit unequal error burdens and students face unequal stigma or support. Advising staff need error-profile literacy, not only risk scores.
Procurement
What should contracts, pricing and exit terms secure?
Write contracts requiring feature/label/objective transparency, group-wise error reporting, and institutional rights to change or exit when post-hoc fairness only redistributes harm.
Operating model
Which teams own the service once it runs?
Pair any EWS go-live with group-wise false-positive/negative reviews and human override paths; do not equate fairness-dashboard activity with equity outcomes.
Publication history
- 2026-10-01Student Success · Issue 253 resources
Stable resource ID: mcconvey-fairness-theatre-ews-20260929