From the Student Success edition of October 4, 2026
Georgia State/NISS AI writing-feedback pilot: 8.1% vs 38.9% DFW for users vs non-users, with self-selection caveat
National Institute for Student Success (NISS) at Georgia State University, in partnership with Studiosity · Higher education academic support / AI-enabled formative writing feedback · United States (Georgia State University, Atlanta)
- Publisher
- National Institute for Student Success (Georgia State University)
- Original publication
- NISS Incubator post dated 30 September 2026 (pilot ran Spring and Fall 2025)
- Source retrieved
- 2026-10-05
What happened
Georgia State faculty integrated Studiosity's LMS-embedded AI writing feedback into courses in Spring and Fall 2025; the expanded fall pilot covered 18 faculty and 1,168 undergraduates, with 71% reported as using the tool. The platform logged more than 3,400 interactions delivered to 942 students; over 80% of engagement fell outside 9 a.m.–5 p.m., peaking at 11 p.m. and on Sundays. Among evaluated writing samples, 66.7% improved across drafts. Users had an 8.1% DFW rate versus 38.9% for non-users (30.8 points). NISS notes participation was voluntary and results may reflect self-selection; faculty cautioned that polish did not always translate into deeper analysis or disciplinary reasoning.
Why it matters
Fresh U.S. public-university student-success signal on AI formative feedback with a DFW outcome, published days after the UMD RCT showing tutor access lowered grades. It is the kind of headline campuses will cite in procurement, so the confounding caveat matters as much as the number.
Evidence and measured results
Descriptive pilot comparison of voluntary users vs non-users (no randomization or covariate adjustment reported); platform usage logs; evaluated writing samples (sample size and rubric not reported); student survey (92% easy to use); faculty feedback. Baseline: non-user DFW in the same participating courses.
Limitations and uncertainty
Voluntary use and self-selection acknowledged by NISS; no adjustment for prior achievement; writing-sample evaluation method and N not stated; figures mix spring and fall (71% of 1,168 vs 942 students reached are not reconciled in the post); vendor partner involved; blog-level reporting, not a full report.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-05; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- High DFW rates in writing-intensive gateway courses and limited human writing support outside business hours.
- Stakeholders
- Student success leadership
- Provost / academic affairs
- Writing center director
- Faculty in gateway writing courses
- Institutional research
- Discovery questions
- Which gateway courses have the highest DFW rates and a drafting assignment structure?
- Can institutional research match AI-feedback users to non-users on prior GPA and course section?
- What share of students need support after 9 p.m. or on weekends?
- Value hypothesis
- Embedded AI writing feedback may lower DFW in drafting-based courses, but the size of any effect is unknown until it is measured against a comparison group.
- Potential engagement
- One-term embedded pilot in 3–5 gateway courses with a pre-registered matched comparison and IR-led outcome analysis.
- Unsupported claims
- That AI feedback causes a 30-point DFW reduction
- That gains transfer to disciplinary reasoning, not just polish
- Faculty grading-time savings
Pre-sales engineering
Role takeaway
- Fit
- LMS-embedded formative writing feedback for undergraduate courses with iterative drafts.
- Architecture and integration
- LTI/LMS embed → vendor feedback service → draft and usage export → IR warehouse joined to SIS outcomes.
- Prerequisites
- LMS integration and SSO
- Data-sharing agreement allowing usage export
- SIS/IR linkage for outcome analysis
- Constraints
- Vendor-hosted processing of student drafts
- FERPA and state privacy obligations
- Off-hours peak load
- Security
- Classify drafts as education records; contract out model training on student work; restrict retention and access.
- Proof of value
- Covariate-adjusted DFW comparison by course
- Rubric-scored revision quality on a blind sample
- Faculty correction-time log
Delivery
Role takeaway
- Work
- Select drafting assignments, train faculty, configure the tool, set the comparison design, and run termly outcome reviews.
- Dependencies
- Faculty willing to embed tool in assignments
- IR capacity for matched analysis
- Vendor data export
- Ownership
- Student success office with the teaching and learning center; IR owns evaluation.
- Skills and adoption
- Faculty set expectations for how students use feedback; students practice revising from feedback rather than accepting edits.
- Governance checkpoints
- Privacy and contract review before launch
- Pre-registered comparison plan
- End-of-term adjusted outcome review before scaling
- Acceptance criteria
- Adjusted DFW difference reported with confidence interval
- At least a defined share of drafts show rubric-scored improvement
- Faculty correction effort documented per course
- Risks
- Self-selection mistaken for impact
- Polish without deeper reasoning
- Vendor lock-in on data access
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
LMS-embedded feedback service needs per-course enablement, draft-version capture to measure revision, and off-hours availability; usage telemetry should be exportable to institutional analytics so user/non-user comparisons can be adjusted for prior GPA and course.
Governance
Who approves, reviews and stays accountable for outcomes?
Treat the 30.8-point DFW gap as unadjusted and confounded; require a matched or randomized comparison before scaling claims. Set assignment-level expectations for how students engage with feedback, as NISS recommends.
Security and privacy
What data, permissions and controls need testing?
Student drafts and feedback are education records processed by a third-party platform; contracts need FERPA terms, retention limits, and clarity on whether drafts train models or feed integrity checks.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Off-hours peaks suggest value for working and caregiving students; faculty still had to manage process and correct the tool, so workload savings are not established.
Procurement
What should contracts, pricing and exit terms secure?
Ask vendors for raw user/non-user outcome data with prior-GPA controls, draft-level revision metrics, and the evaluation rubric before accepting DFW claims in RFP responses.
Operating model
Which teams own the service once it runs?
Run AI writing feedback as an embedded course practice with faculty training and assignment alignment, owned jointly by student-success and teaching-and-learning units, with a planned comparison design.
Publication history
- 2026-10-04Student Success · Issue 284 resources
Stable resource ID: gsu-niss-studiosity-ai-writing-feedback-pilot-20260930