{"resourceId":"gsu-niss-studiosity-ai-writing-feedback-pilot-20260930","versions":[{"version":"external-6b418884b46da06fc6084f6a7276ec0eceb910edb1052b2b80cc12e448955bc9","resource":{"id":"gsu-niss-studiosity-ai-writing-feedback-pilot-20260930","title":"Georgia State/NISS AI writing-feedback pilot: 8.1% vs 38.9% DFW for users vs non-users, with self-selection caveat","organization":"National Institute for Student Success (NISS) at Georgia State University, in partnership with Studiosity","sector":"Higher education academic support / AI-enabled formative writing feedback","geography":"United States (Georgia State University, Atlanta)","publishedAt":"NISS Incubator post dated 30 September 2026 (pilot ran Spring and Fall 2025)","publicationDate":"2026-09-30","eventDate":null,"sourceName":"National Institute for Student Success (Georgia State University)","sourceLabel":"University institute pilot write-up (vendor-partnered)","sourceUrl":"https://niss.gsu.edu/2026/09/30/early-findings-from-an-a-i-writing-assistant-pilot/","evidenceClass":"independent-research","outcomeClass":"emerging","topics":["knowledge-work","operating-model","accessibility-workforce","governance-procurement"],"finding":"Georgia State faculty integrated Studiosity's LMS-embedded AI writing feedback into courses in Spring and Fall 2025; the expanded fall pilot covered 18 faculty and 1,168 undergraduates, with 71% reported as using the tool. The platform logged more than 3,400 interactions delivered to 942 students; over 80% of engagement fell outside 9 a.m.–5 p.m., peaking at 11 p.m. and on Sundays. Among evaluated writing samples, 66.7% improved across drafts. Users had an 8.1% DFW rate versus 38.9% for non-users (30.8 points). NISS notes participation was voluntary and results may reflect self-selection; faculty cautioned that polish did not always translate into deeper analysis or disciplinary reasoning.","sledRelevance":"Fresh U.S. public-university student-success signal on AI formative feedback with a DFW outcome, published days after the UMD RCT showing tutor access lowered grades. It is the kind of headline campuses will cite in procurement, so the confounding caveat matters as much as the number.","evidence":"Descriptive pilot comparison of voluntary users vs non-users (no randomization or covariate adjustment reported); platform usage logs; evaluated writing samples (sample size and rubric not reported); student survey (92% easy to use); faculty feedback. Baseline: non-user DFW in the same participating courses.","architectureImplications":"LMS-embedded feedback service needs per-course enablement, draft-version capture to measure revision, and off-hours availability; usage telemetry should be exportable to institutional analytics so user/non-user comparisons can be adjusted for prior GPA and course.","governanceImplications":"Treat the 30.8-point DFW gap as unadjusted and confounded; require a matched or randomized comparison before scaling claims. Set assignment-level expectations for how students engage with feedback, as NISS recommends.","securityPrivacyImplications":"Student drafts and feedback are education records processed by a third-party platform; contracts need FERPA terms, retention limits, and clarity on whether drafts train models or feed integrity checks.","caveats":"Voluntary use and self-selection acknowledged by NISS; no adjustment for prior achievement; writing-sample evaluation method and N not stated; figures mix spring and fall (71% of 1,168 vs 942 students reached are not reconciled in the post); vendor partner involved; blog-level reporting, not a full report.","streamIds":["student-success"],"roles":{"sales":"The opening point is that DFW reduction is the outcome provosts buy, and this pilot reports a large gap: 8.1% versus 38.9%. Use it to start a conversation about writing support for working students who study late at night, not as proof of impact. The comparison is voluntary users against non-users, so position any engagement as a measured pilot that produces adjusted evidence. Never quote the 30.8-point gap as an expected result for a new campus.","engineering":"Engineering's framing point is that the outcome can't be trusted without the data plumbing. An LMS-embedded feedback service needs draft versioning, per-student usage export, and linkage to the student information system so analysts can compare users and non-users after controlling for prior GPA, course, and section. Plan for off-hours load, since 80% of use fell outside business hours. Review vendor data flows for FERPA terms and model-training exclusions before any integration work.","delivery":"Delivery should treat this as a design pattern, not a result. Embed the tool in specific drafting assignments with faculty training and clear expectations, as NISS recommends. Define the comparison group before launch, track DFW rate, revision quality, and faculty correction effort, and report adjusted results each term. Watch for students polishing prose without improving their reasoning, which faculty flagged. Decide to scale only after a matched or randomized comparison."},"roleDetails":{"sales":{"customerProblem":"High DFW rates in writing-intensive gateway courses and limited human writing support outside business hours.","stakeholders":["Student success leadership","Provost / academic affairs","Writing center director","Faculty in gateway writing courses","Institutional research"],"discoveryQuestions":["Which gateway courses have the highest DFW rates and a drafting assignment structure?","Can institutional research match AI-feedback users to non-users on prior GPA and course section?","What share of students need support after 9 p.m. or on weekends?"],"valueHypothesis":"Embedded AI writing feedback may lower DFW in drafting-based courses, but the size of any effect is unknown until it is measured against a comparison group.","potentialEngagement":"One-term embedded pilot in 3–5 gateway courses with a pre-registered matched comparison and IR-led outcome analysis.","unsupportedClaims":["That AI feedback causes a 30-point DFW reduction","That gains transfer to disciplinary reasoning, not just polish","Faculty grading-time savings"]},"engineering":{"fit":"LMS-embedded formative writing feedback for undergraduate courses with iterative drafts.","architectureIntegration":"LTI/LMS embed → vendor feedback service → draft and usage export → IR warehouse joined to SIS outcomes.","prerequisites":["LMS integration and SSO","Data-sharing agreement allowing usage export","SIS/IR linkage for outcome analysis"],"constraints":["Vendor-hosted processing of student drafts","FERPA and state privacy obligations","Off-hours peak load"],"security":"Classify drafts as education records; contract out model training on student work; restrict retention and access.","proofOfValue":["Covariate-adjusted DFW comparison by course","Rubric-scored revision quality on a blind sample","Faculty correction-time log"]},"delivery":{"work":"Select drafting assignments, train faculty, configure the tool, set the comparison design, and run termly outcome reviews.","dependencies":["Faculty willing to embed tool in assignments","IR capacity for matched analysis","Vendor data export"],"ownership":"Student success office with the teaching and learning center; IR owns evaluation.","skillsAdoption":"Faculty set expectations for how students use feedback; students practice revising from feedback rather than accepting edits.","governanceCheckpoints":["Privacy and contract review before launch","Pre-registered comparison plan","End-of-term adjusted outcome review before scaling"],"acceptanceCriteria":["Adjusted DFW difference reported with confidence interval","At least a defined share of drafts show rubric-scored improvement","Faculty correction effort documented per course"],"risks":["Self-selection mistaken for impact","Polish without deeper reasoning","Vendor lock-in on data access"]}},"retrievedAt":"2026-10-05T03:17:35Z","enrichedAt":"2026-10-05T03:21:28Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Off-hours peaks suggest value for working and caregiving students; faculty still had to manage process and correct the tool, so workload savings are not established.","procurementImplications":"Ask vendors for raw user/non-user outcome data with prior-GPA controls, draft-level revision metrics, and the evaluation rubric before accepting DFW claims in RFP responses.","operatingModelImplications":"Run AI writing feedback as an embedded course practice with faculty training and assignment alignment, owned jointly by student-success and teaching-and-learning units, with a planned comparison design.","sourceVerification":{"openedUrl":"https://niss.gsu.edu/2026/09/30/early-findings-from-an-a-i-writing-assistant-pilot/","referenceExcerpt":"Students who engaged with Studiosity had a DFW rate of 8.1%, compared with 38.9% among non-users, a difference of 30.8 percentage points.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}