From the Student Success edition of October 4, 2026
Lethbridge CourseChat: on-premises multi-course RAG tutor keeps an 8B model after larger models failed a classroom speed gate
Dhillon School of Business, University of Lethbridge (Shapiro, Lindemann) · Higher education business courses / campus-hosted GenAI tutoring infrastructure · Canada (University of Lethbridge, Alberta)
- Publisher
- arXiv
- Original publication
- Submitted to arXiv 1 October 2026 (cs.AI)
- Source retrieved
- 2026-10-05
What happened
CourseChat serves six isolated course offerings (keyed by CRN) from twin on-premises DGX Spark-class hosts running FastAPI, Qdrant, local embeddings and reranking, and Ollama, behind a campus IIS gateway for Moodle embedding. Several larger models failed a speed gate (median latency no more than 30% above the 8B baseline); a 12B and a 7B model passed, but the team kept llama3.1:8b pending a demonstrated overall improvement. In a five-course replay, all 400 scope checks passed, each course hit the 90% follow-up target, and median per-turn latency ran 3.24–6.19 s; 93 no-match turns remained. 435 prebuilt quiz questions across 65 modules separate practice from live generation. The authors state no learning gains are established.
Why it matters
Campuses weighing a course-grounded tutor that keeps textbooks and student dialogue on institutional hardware get a concrete reference architecture and its limits, a useful counterpoint to vendor-hosted tutors after the UMD RCT showed unstructured access can backfire.
Evidence and measured results
Engineering evaluations: two model bake-off rounds with fixed retrieval, a source-fidelity comparison, frozen 20-dialogue/80-turn fixtures per course, quiz audits; serial server-local latency; no load study, no faculty ratings, no learning-outcome study.
Limitations and uncertainty
Single business school; engineering regressions are not a representative sample of student use; latency measurements serial; peak-load capacity untested; faculty ratings pending; no learning outcomes.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-05; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions want course-grounded AI tutoring without sending textbooks or student dialogue to external providers.
- Stakeholders
- CIO / research computing
- Business school dean
- Faculty course leads
- Privacy officer
- Procurement
- Discovery questions
- Which courses have licensed materials that cannot leave campus?
- What concurrency do you expect at peak class times?
- How will this add value beyond existing Copilot/Gemini licenses?
- Value hypothesis
- An on-premises, course-isolated RAG tutor can provide grounded help with institutional data control, if latency and faculty workload are managed.
- Potential engagement
- Architecture review and load-tested pilot for 2–3 courses with faculty-rated answer quality.
- Unsupported claims
- Learning gains
- Classroom-scale capacity
- That a larger model would improve tutoring
Pre-sales engineering
Role takeaway
- Fit
- Course-grounded tutoring for a defined set of courses with institution-owned materials.
- Architecture and integration
- Campus gateway (TLS, routing) → replicated nodes with FastAPI, vector DB, embeddings/reranker, local LLM → Moodle embed.
- Prerequisites
- On-premises GPU appliances with unified memory
- Course corpus ingestion pipeline
- Gateway SSE support
- Constraints
- Serial-only latency evidence
- Cold-load delays for large models
- Per-node artifact replication
- Security
- Keep data local, enforce per-CRN collection isolation, and verify every public route before acceptance.
- Proof of value
- Concurrent browser-to-gateway latency under representative load
- False-refusal and source-error rates on held-out dialogues
- Faculty usefulness ratings
Delivery
Role takeaway
- Work
- Ingest course materials, build and review quiz banks, run regression fixtures, and operate change control for model updates.
- Dependencies
- Faculty content review time
- Local MLOps staff
- Gateway configuration
- Ownership
- Central or school IT for the platform; faculty for course content.
- Skills and adoption
- Faculty learn corpus curation and question review; students are guided to attempt problems before seeking hints.
- Governance checkpoints
- Model promotion rule met before swap
- Course isolation test each term
- Privacy review of logging
- Acceptance criteria
- Median latency within agreed target at expected concurrency
- Defined follow-up topic-hit rate per course
- Faculty-rated accuracy threshold on held-out set
- Risks
- Faculty workload underestimated
- Capacity shortfall at peak
- No measured learning benefit
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Per-course isolated vector collections, replicated local stacks per node, retrieval-topic resolution for follow-ups, bounded evidence selection, and prebuilt practice banks; model choice is a joint decision with retrieval, serving compatibility, and latency.
Governance
Who approves, reviews and stays accountable for outcomes?
Promotion rules for model changes (speed gate, scope preservation, reviewed improvement) are a reusable change-control pattern; faculty must own corpus content and question review.
Security and privacy
What data, permissions and controls need testing?
On-premises hosting keeps copyrighted textbooks and student dialogue off external APIs; strict course isolation prevents cross-course leakage; gateway acceptance for every course slug was still open.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Requires local MLOps skills and faculty time to prepare materials, review questions, and fix errors between offerings; authors propose tracking that workload.
Procurement
What should contracts, pricing and exit terms secure?
On-premises appliance purchases should be scoped against measured concurrency and latency under load, not memory alone; compare against already-licensed Copilot/Gemini access.
Operating model
Which teams own the service once it runs?
Run as a joint IT and faculty service with change control for model swaps, course-corpus refresh each term, and a student flagging loop feeding regression cases.
Publication history
- 2026-10-04Student Success · Issue 284 resources
Stable resource ID: lethbridge-coursechat-onprem-rag-tutor-20261001