{"resourceId":"duke-ctl-coursera-dialogue-ai-coach-pilot-20261102","versions":[{"version":"external-9ec353b069dfe9e2670293523d4dcc28301bc48bcfaa0b495ab8bdaf578a73e3","resource":{"id":"duke-ctl-coursera-dialogue-ai-coach-pilot-20261102","title":"Duke CTL (2 Nov): six-course pilot of Coursera's Dialogue AI coach gets about 84% would-use-again among respondents, but needed heavy authoring and framing and split learners","organization":"Duke University Center for Teaching and Learning","sector":"Higher education online and MOOC instruction","geography":"United States (North Carolina)","publishedAt":"Duke Center for Teaching and Learning blog, 2 November 2026 (updated 4 November)","publicationDate":"2026-11-02","eventDate":null,"sourceName":"Duke Center for Teaching and Learning","sourceLabel":"Institutional pilot report (vendor tool, operator-authored)","sourceUrl":"https://ctl.duke.edu/blog/2026/11/does-ai-deliver-observing-the-effects-of-interactive-learning-experiences-with-an-ai-powered-chatbot/","evidenceClass":"vendor-claim","outcomeClass":"mixed","topics":["knowledge-work","operating-model","accessibility-workforce"],"finding":"Duke's Learning Experience Design team piloted Coursera's Dialogue tool, which has learners hold ungraded, real-time conversations with an AI \"Coach\" built from course material and instructor input, in six self-paced Coursera courses. About 84 percent of learners who engaged with Dialogue and took the optional survey somewhat or strongly agreed they would use it again. Open-ended responses varied widely, from wholesale endorsement to refusal, with ethical qualms, overreliance and privacy concerns alongside appreciation for interactivity. The authors say the activities took significantly more time to author and test than a typical assessment and needed extra scaffolding; an introductory reading that explained the pilot's values measurably improved learner perception. They call the pilot a success, say AI tools are \"far from a silver bullet,\" and note long-term learning effects remain unanswered.","sledRelevance":"Shows an elite university piloting a vendor AI coach at MOOC scale, with a candid account of authoring cost, learner variation and framing, and with perception rather than learning outcomes as the success metric.","evidence":"Operator-authored summary of an optional learner survey in six courses; no denominators, comparison group or learning-outcome data are reported, and the tool is from the platform partner.","architectureImplications":"A vendor-hosted conversational coach embedded in a course platform, configured per activity from course material and instructor inputs; no data architecture detail is given.","governanceImplications":"Instructor co-design and approval gate each activity, and pre-reading explains the institution's values on AI use; learners could choose how to engage.","securityPrivacyImplications":"Learners raised privacy concerns; the article does not detail data handling by the vendor.","caveats":"Self-selected survey respondents; satisfaction rather than learning measured; partner-vendor tool; MOOC context may not transfer to credit courses.","streamIds":["student-success"],"roles":{"sales":"Sales framing: online and large-enrollment programs want interactive practice at scale, and Duke reports an AI coach that learners mostly would reuse. The practical lesson is that authoring time, framing and learner choice decide success. A bounded engagement is a pilot design and authoring-workflow review for a few courses. The evidence is an operator-written, optional survey with about 84% approval among respondents; it cannot support learning-gain, retention or cost-saving claims.","engineering":"Engineering framing: fit is a platform-embedded AI coach configured per activity from course content and instructor input. Check how conversation data are stored and who can access them, how guardrails and accuracy are tested, and how activities are versioned. Duke notes substantial testing before release. Validate with pre-launch conversation testing, a no-AI alternative path, and instrumentation to compare completion and assessment results with non-users.","delivery":"Delivery framing: Duke's own account shows delivery effort sits in authoring, testing and learner framing. Plan instructor co-design, a pre-reading that explains why AI is used, and an opt-out or alternative for learners who object. Governance checkpoints cover instructor approval and learner feedback review. Acceptance should be measurable completion, response quality and learning checks, since the pilot measured perceptions only."},"roleDetails":{"sales":{"customerProblem":"Large asynchronous courses struggle to give individualized feedback and practice at scale.","stakeholders":["Online learning leadership","Course instructors","Instructional designers","Platform partner","Learners"],"discoveryQuestions":["How much authoring time per AI activity is acceptable?","What learning measures beyond satisfaction exist?","What alternative do objecting learners get?"],"valueHypothesis":"Interpretation: well-framed AI practice activities may add low-stakes interaction at scale; the source reports perceived value only.","potentialEngagement":"Pilot design for two or three courses: authoring template, learner framing, opt-out path and learning-outcome measurement.","unsupportedClaims":["Learning gains","Retention or completion improvement","Cost savings"]},"engineering":{"fit":"Vendor AI coach embedded in an online course platform.","architectureIntegration":"Course-platform embedding, instructor-supplied content and prompts, survey capture and optional learner analytics.","prerequisites":["Course content permissions","Platform partner configuration","Conversation testing protocol"],"constraints":["Vendor-hosted tool","No data handling detail in source","MOOC context"],"security":"Establish what conversation data the vendor retains and how it is used before scaling.","proofOfValue":["Pre-release adversarial testing of conversations","A/B completion and quiz comparison","Survey with reported denominators"]},"delivery":{"work":"Co-design AI activities with instructors, test them, write learner framing, and review feedback.","dependencies":["Instructors","Instructional designers","Platform partner","Privacy office"],"ownership":"Teaching and learning center with instructors","skillsAdoption":"Instructors need examples and templates for AI-supported activity design; learners need explanation of how and why AI is used.","governanceCheckpoints":["Instructor approval of each activity","Privacy review","Post-term review of learner feedback"],"acceptanceCriteria":["Proposed: authoring hours per activity tracked","Proposed: completion and quiz comparison with non-users","Proposed: survey response rate and denominators reported"],"risks":["Authoring cost outweighs benefit","Learners disengage over AI ethics concerns","Satisfaction mistaken for learning"]}},"retrievedAt":"2026-11-11T04:15:57Z","enrichedAt":"2026-11-11T04:19:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Authoring effort burdens instructional designers and faculty; some learners will disengage entirely, so an alternative path matters.","procurementImplications":"Embedded in an existing platform partnership; cost is mostly design labor, which procurement conversations should surface.","operatingModelImplications":"A central teaching center co-designs with faculty and the platform vendor, using learner and instructor feedback as pilot metrics.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://ctl.duke.edu/blog/2026/11/does-ai-deliver-observing-the-effects-of-interactive-learning-experiences-with-an-ai-powered-chatbot/","referenceExcerpt":"These activities required significantly more time to author and test than a typical assessment, and additional course scaffolding to introduce them to learners.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}