{"resourceId":"ajpe-utoronto-course-trained-ai-virtual-tutor-pharmacology-202-students-20261102","versions":[{"version":"external-5976951670c6227122c4c54c97b292f111d91a6c0d2ba1caa675c6af5ff16534","resource":{"id":"ajpe-utoronto-course-trained-ai-virtual-tutor-pharmacology-202-students-20261102","title":"University of Toronto course-trained AI tutor in second-year pharmacology: high uptake, ~80% fewer discussion-board content queries, but no grade difference between user groups","organization":"University of Toronto (Department of Pharmacology and Toxicology; Information Technology Services; Centre for Teaching Support and Innovation)","sector":"Higher education teaching and student academic support","geography":"Canada (Toronto)","publishedAt":"American Journal of Pharmaceutical Education, published online 2 November 2026","publicationDate":"2026-11-02","eventDate":null,"sourceName":"American Journal of Pharmaceutical Education","sourceLabel":"Peer-reviewed journal article (open access, DOI 10.1016/j.ajpe.2026.102120)","sourceUrl":"https://www.ajpe.org/article/S0002-9459(26)01478-6/fulltext","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","operating-model","governance-procurement"],"finding":"A University of Toronto team integrated an in-house, retrieval-augmented AI virtual tutor trained on course content into Canvas for a large second-year undergraduate pharmacology course (published online 2 November 2026). Of 244 students, 202 (83%) completed the survey; 234 users accessed the tutor module and 141 returned. The tutor logged 3,379 interactions, mostly between 5 pm and 3 am and spiking before tests. Course discussion-board content queries fell about 80% relative to the prior year. Frequent users reported more positive views of learning, resilience and confidence. Final grades did not differ significantly between user groups; rare users had the highest average (81.5%), and the authors could not link tutor use to grade improvement.","sledRelevance":"Gives student-success leaders measured usage and support-load data for a course-specific AI tutor, plus a null grade result that tempers claims about AI tutors improving outcomes.","evidence":"Mixed-methods: Likert and open-ended survey (n=202), LMS access logs, and coded conversation data; ethics approval reported; Kruskal-Wallis and ANOVA comparisons.","architectureImplications":"In-house RAG tutor trained on lecture notes and syllabi, hosted in the institutional Canvas LMS; access was removed during weeks 9-12 for major assignments.","governanceImplications":"Interpretation: course-scoped tutors let instructors control content sources and timing, which supports assessment integrity choices.","securityPrivacyImplications":"Conversation data were anonymized for analysis; survey data were de-identified after grades were submitted.","caveats":"Single course; self-report bias acknowledged; no control group; Likert themes grouped with help from Perplexity AI; the discussion-board drop may have other causes.","streamIds":["student-success"],"roles":{"sales":"Sales framing: course-specific AI tutors attract heavy use and can shift routine questions away from staff. In this Toronto course, 234 users opened the tutor and discussion-board content queries fell about 80%, but grades did not differ between user groups. A credible pitch is about after-hours support and teaching insight, not grade gains. A bounded offer is a one-course tutor pilot with usage and outcome evaluation. Do not claim AI tutors raise grades; this study found no significant difference.","engineering":"Engineering framing: this was an in-house retrieval-augmented tutor trained on course notes and syllabi and hosted in Canvas. Fit depends on LMS integration, a controlled content corpus, and the ability to switch access off during assessed work, as the team did in weeks 9 to 12. Prerequisites include content preparation and logging. Proof of value should pair usage logs with grade data and a comparison group, since this study relied on self-report and a prior-year baseline.","delivery":"Delivery framing: the operating owner is the course team with central IT and the teaching centre. Work includes preparing course content for retrieval, setting access windows, coaching students on prompting, and reviewing conversation analytics to spot hard topics. Acceptance should rest on measurable usage, support-load and grade comparisons agreed in advance. Risks are overclaiming learning gains, students who distrust AI losing human support, and accuracy issues that students flagged in open-ended responses."},"roleDetails":{"sales":{"customerProblem":"Large courses cannot give students timely after-hours help, and staff field repetitive content questions.","stakeholders":["Course instructors","Teaching and learning centre","Central IT","Associate dean for undergraduate education"],"discoveryQuestions":["When do your students ask for help, and who answers?","Which course content could ground a tutor?","What outcome would justify scaling?"],"valueHypothesis":"Interpretation: a course-scoped tutor can absorb routine content questions and reveal hard topics, without proven grade gains.","potentialEngagement":"A one-course tutor pilot with usage, support-load and grade evaluation against a comparison group.","unsupportedClaims":["AI tutors improve grades","The 80% board-query drop is caused by the tutor alone","Results generalise beyond this course"]},"engineering":{"fit":"Course-scoped RAG tutor inside the LMS.","architectureIntegration":"Canvas-hosted tutor grounded on lecture notes and syllabi; access windows controllable by instructors; interaction logs exported for analysis.","prerequisites":["Curated course content","LMS integration","Logging and anonymization pipeline"],"constraints":["Single-course evidence","Tutor could not read graphs or images well per student feedback","Model accuracy concerns"],"security":"Anonymize conversation logs; define retention; keep student prompts out of external model training.","proofOfValue":["Grade comparison with a control section","Discussion-board and office-hours load versus baseline","Answer accuracy audit on sampled conversations"]},"delivery":{"work":"Stand up a course tutor, coach students on prompting, and review analytics each term.","dependencies":["Instructor time for content curation","IT hosting","Ethics or data-governance approval for analytics"],"ownership":"Course team with central IT and the teaching centre.","skillsAdoption":"Students need prompting guidance; instructors need to read conversation analytics.","governanceCheckpoints":["Access windows set around assessments","Data-use approval","End-of-term evaluation review"],"acceptanceCriteria":["Proposed: grade comparison reported with a comparison group","Proposed: human help channels remain available","Proposed: sampled answer accuracy reviewed"],"risks":["Overclaiming learning gains","Reduced human contact for some students","Inaccurate answers"]}},"retrievedAt":"2026-11-03T04:12:30Z","enrichedAt":"2026-11-03T04:14:36Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Most use was after hours when staff are unavailable; reduced board queries may lower instructor workload, but students who avoid AI still need human channels.","procurementImplications":"Interpretation: built in-house rather than purchased, which avoids per-seat costs but needs IT and teaching-centre capacity.","operatingModelImplications":"Interpretation: conversation analytics showed which topics students struggled with, giving instructors a feedback loop for targeted teaching.","peerReview":"peer-reviewed","sourceVerification":{"openedUrl":"https://www.ajpe.org/article/S0002-9459(26)01478-6/fulltext","referenceExcerpt":"there was an approximate 80% reduction in student queries on the course discussion board for course content questions","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}