{"resourceId":"yang-structured-tutor-modality-mba-rct-20260921","versions":[{"version":"external-7a715863c2362fdf022d9b0815af1a0994b730ec0913bcb4542ad9df0b61db58","resource":{"id":"yang-structured-tutor-modality-mba-rct-20260921","title":"BU online-MBA RCT: structured course-grounded AI tutor beats ambient GenAI; voice≠mastery","organization":"Boston University Questrom School of Business","sector":"Higher education / academic research","geography":"United States (online MBA corporate-finance module; transferable with limits)","publishedAt":"2026-09-21","publicationDate":"2026-09-21","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint (arXiv:2609.23958)","sourceUrl":"https://arxiv.org/pdf/2609.23958","evidenceClass":"academic-research","outcomeClass":"effective","topics":["knowledge-work","developers-agents","operating-model","accessibility-workforce"],"finding":"In a preregistered field RCT (N=86 consented), a structured tutor grounded in course materials raised scores 6.63 points over ability-matched peers with free consumer AI (p=.007), with gains concentrated in relational written reasoning; within-student voice vs text yielded equivalent weekly mastery at 2.8× voice cost. Why now: submitted 2026-09-21 and absent from the archive after the 2026-09-21 edition.","sledRelevance":"Directly informs U.S. HE tutoring and online graduate programs that pedagogical structure—not voice anthropomorphism—drives learning against the realistic counterfactual of ambient ChatGPT-like tools.","evidence":"Preregistered OSF plan; between-arm ANCOVA and instructor gradebook on intact randomization; within-student modality crossover with billing-based cost accounting. Single MBA module.","architectureImplications":"Prefer course-RAG tutors with ask-before-answer scaffolds; treat voice as optional engagement layer with explicit cost controls; log modality separately from mastery.","governanceImplications":"Require evaluation designs that compare structured tutors to ambient AI holdouts, not to no-AI fantasies; disclose modality cost–benefit to procurement.","securityPrivacyImplications":"Course-grounded tutors still process student dialogue—keep sessions in approved tenancy with retention limits.","caveats":"Online MBA finance context; self-selected consent; N=51 for dual assessments; Hawthorne possible.","streamIds":["student-success"],"roles":{"sales":"For teaching-and-learning and online program buyers, this BU field RCT shows a pedagogically structured, course-grounded AI tutor improves relational reasoning over students who already use consumer AI freely—while voice nearly doubles interaction cost without raising mastery. Position discovery around structure, assessment of written reasoning depth, and cost of modality—not anthropomorphic demos. Bounded value is designing and evaluating structured tutors against ambient AI baselines, not guaranteeing GPA lifts from voice assistants.","engineering":"Engineering takeaway: implement course-material RAG with tutor guardrails, dual text/voice channels, and mastery instruments that score relational written answers—not only MCQ. Prerequisites include preregistered outcomes, LMS/auth integration, and provider billing telemetry. Constraints: single MBA module evidence; opt-in sample. Security: education-record treatment of transcripts. Proof of value: reproduce structured vs ambient gap and voice–text mastery equivalence locally before buying voice premium.","delivery":"Delivery work is a course pilot: structured tutor vs ambient holdout, with relational rubrics and optional voice weeks. Dependencies: faculty author of course corpus, academic technology, IR/ethics. Ownership: program director with teaching-and-learning. Skills: rubric training for relational quality. Governance checkpoints: syllabus AI policy and cost review. Acceptance: documented structured-tutor advantage on reasoning depth; risks include over-buying voice for engagement alone."},"roleDetails":{"sales":{"customerProblem":"Campuses equate voice GenAI tutors with better learning and under-invest in pedagogical structure versus ambient consumer AI.","stakeholders":["online/graduate program deans","teaching & learning","academic technology","student success"],"discoveryQuestions":["Do you evaluate tutors against ambient consumer AI holdouts?","How do you score relational written reasoning?","What is the cost delta for voice vs text tutoring?"],"valueHypothesis":"Bounded value: design structured course-grounded tutors and measure reasoning gains before funding voice anthropomorphism.","potentialEngagement":"6–8 week structured-tutor pilot with ambient-AI control and relational rubric—not an outcome guarantee.","unsupportedClaims":["Voice GenAI always improves learning","Guaranteed MBA grade gains from any AI tutor","Transfer of online MBA finance results to all undergrad courses"]},"engineering":{"fit":"Fits as course-grounded tutoring layer with optional voice channel atop approved LLM/RAG stack.","architectureIntegration":"Corpus RAG + tutor policy prompts + LMS SSO; optional STT/TTS cascade; mastery quiz bank.","prerequisites":["Approved course materials corpus","Relational scoring rubric","Provider cost telemetry"],"constraints":["Single-module external evidence","Consent self-selection","Graduate online setting"],"security":"Treat dialogue transcripts as education records; disable training on student data without DPA.","proofOfValue":["Structured vs ambient score gap","Voice–text mastery equivalence","Cost per mastery point by modality"]},"delivery":{"work":"Stand up structured tutor, randomize or match holdout, score relational writing, review voice costs.","dependencies":["Faculty SMEs","Academic technology","Assessment analytics"],"ownership":"Course lead jointly with teaching & learning / academic technology.","skillsAdoption":"Train faculty/TAs on relational rubrics and tutor prompt governance.","governanceCheckpoints":["Syllabus AI disclosure","Procurement cost review for voice","IRB/ethics if research-framed"],"acceptanceCriteria":["Documented structured-tutor advantage on reasoning depth","Published modality cost–benefit note"],"risks":["Hawthorne inflation","Over-interpreting MBA results for community colleges","Voice spend without mastery lift"]}},"retrievedAt":"2026-09-23T03:18:16Z","enrichedAt":"2026-09-23T03:24:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Voice may aid some accessibility preferences but did not improve mastery here; keep text parity and accommodations.","procurementImplications":"Separate line items for structured tutor design vs voice premium; require ambient-AI baseline evaluations in RFPs.","operatingModelImplications":"Shift operating KPI from chat volume/NPS to relational reasoning gains and cost per mastery point.","sourceVerification":{"responseId":"curl-arxiv-2609-23958","openedUrl":"https://arxiv.org/pdf/2609.23958","referenceExcerpt":"Structure mattered: tutored students gained 6.63 points more than ability-matched peers (p = .007)","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}