{"resourceId":"gray-hobbs-beacon-rag-helpseeking-20260918","versions":[{"version":"external-0558b6cecb290840924050e5d151e3a7bc1829ab9d7870f196fe56012f155397","resource":{"id":"gray-hobbs-beacon-rag-helpseeking-20260918","title":"Beacon course-specific RAG: students trust module-aligned help over unrestricted GenAI; outcomes unmeasured","organization":"Bath Spa University","sector":"Higher education / academic research","geography":"United Kingdom (computing undergraduates; transferable with limits)","publishedAt":"2026-09-18","publicationDate":"2026-09-18","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint (arXiv:2609.21600)","sourceUrl":"https://arxiv.org/pdf/2609.21600","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["knowledge-work","infrastructure","accessibility-workforce","operating-model","data-security"],"finding":"Design-based research of Beacon, a course-specific RAG system grounded in approved teaching materials. Main evaluation with 15 computing undergraduates and interviews with five students and four staff found participants viewed responses as module-aligned and more trustworthy than unrestricted GenAI, valuing scaffolding/pseudocode over direct solutions and positioning Beacon as a first step before staff. Why now: posted 2026-09-18; absent from archive; newly relevant amid U.S. early-alert and tutoring-bot rollouts seeking equitable help-seeking.","sledRelevance":"Informs U.S. SLED student-support AI: course-grounded RAG may lower help-seeking barriers without replacing advisors/tutors—but validate learning and equity outcomes, not trust ratings alone.","evidence":"DBR pilot then main study; questionnaires and semi-structured interviews; perception/usability focus; open-source localizable stack claimed. No RCT on grades or persistence.","architectureImplications":"Prefer RAG over approved course corpora with source display, scaffolded responses, and institutional hosting rather than open consumer chat for academic help.","governanceImplications":"Define Beacon-like tools as complementary first-line support with escalation to humans; publish accuracy/limitation notices.","securityPrivacyImplications":"Local/open-source deployment can keep student queries and course materials under institutional control—align with FERPA/vendor DPAs in U.S. ports.","caveats":"N=15 perception study; UK computing only; no causal learning or equity-gap closure evidence.","streamIds":["student-success"],"roles":{"sales":"For advising and tutoring leaders chasing always-on help, Beacon shows students may trust course-specific RAG more than unrestricted GenAI and treat it as a first step before staff—useful for help-seeking equity narratives without replacing humans. Do not sell grade or retention lifts from this paper; evidence is perception-only from fifteen volunteers. Bounded value is a module-aligned RAG pilot with human escalation and a real evaluation plan.","engineering":"Engineering takeaway: implement RAG over approved course materials with visible sources, scaffolded answers such as pseudocode and hints, and institutional hosting rather than open consumer chat. Prerequisites include a curated course corpus, embedding and index operations, and a faculty feedback loop. Constraints: small UK usability sample with no causal learning data. Security: keep queries on campus infrastructure. Proof of value: help-seeking uptake among prior non-help-seekers plus later learning endpoints—not trust scores alone.","delivery":"Delivery work: pick one high-DFW course, curate materials into a RAG assistant, train staff on escalation paths, and run a mixed-methods evaluation with a pre-registered learning endpoint. Ownership rests with academic support plus the course faculty lead. Skills needed include content curation and ongoing RAG operations. Governance covers accuracy review and a student data notice. Acceptance: documented usage metrics and learning results even if null; risks include scaling on trust surveys alone."},"roleDetails":{"sales":{"customerProblem":"Students avoid office hours yet lean on inaccurate consumer GenAI; campuses want private, course-aligned help without replacing humans.","stakeholders":["tutoring / academic support","faculty in gateway courses","student success","IT/security"],"discoveryQuestions":["Which course materials can legally ground a RAG corpus?","What is the human escalation path?","Which learning or equity endpoints are pre-registered?"],"valueHypothesis":"Course-specific RAG can improve perceived access and trust versus consumer GenAI as a first-line layer—learning gains remain unproven here.","potentialEngagement":"8–10 week single-course RAG pilot with evaluation design—not a campus-wide chatbot rollout.","unsupportedClaims":["Proven grade or retention improvement","Closure of help-seeking equity gaps","Replacement of tutors/advisors"]},"engineering":{"fit":"Module-scoped RAG assistant with source citation UI and staff dashboard.","architectureIntegration":"Corpus pipeline from LMS/approved docs → embeddings → LLM generate-with-citations → optional ticket to tutoring.","prerequisites":["Faculty-owned content pack","Hosting decision (on-prem/VPC)","Evaluation instruments"],"constraints":["Perception-only external evidence","Computing-course context","Corpus freshness burden"],"security":"Institutional tenancy for queries and materials; role-based staff access; retention limits.","proofOfValue":["Uptake among low prior help-seekers","Source-alignment spot audits","Pre-registered learning contrast"]},"delivery":{"work":"Corpus build, pilot deploy, student/staff interviews, mid-course content refresh.","dependencies":["Faculty SMEs","Hosting/budget","IR if publishing"],"ownership":"Academic support director with course lead co-owner.","skillsAdoption":"Train students to verify sources and escalate; train staff to treat Beacon as triage not replacement.","governanceCheckpoints":["Data notice","Accuracy review board","Go/no-go after pilot metrics"],"acceptanceCriteria":["Pilot report complete","Escalation SLA defined","No unverified outcome claims in marketing"],"risks":["Hallucinated advice despite RAG","Faculty corpus lag","Students never escalate when needed"]}},"retrievedAt":"2026-09-22T03:12:40Z","enrichedAt":"2026-09-22T03:16:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Private, always-available course help may aid anxious or time-poor students who avoid staff scrutiny, but must meet accessibility standards and not become the only path for students who need human support.","procurementImplications":"Favor vendors/open stacks that ground answers in licensed course content, expose sources, and allow institutional hosting—require evaluation clauses.","operatingModelImplications":"Position course RAG as intermediary between self-study and tutors; protect tutor capacity for complex cases.","sourceVerification":{"responseId":"curl-arxiv-2609-21600","openedUrl":"https://arxiv.org/pdf/2609.21600","referenceExcerpt":"Students consistently described Beacon’s responses as closely aligned with module content and more trustworthy than unrestricted generative AI tools","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}