From the Campus Operations edition of October 4, 2026
Augie Mirror test of Augustana's Gemini-based 'Ask Augustana' chatbot finds stale answers, prompt disclosure and scope drift
Augustana University (South Dakota) / Zion Cloud Solutions (via The Augie Mirror) · Private higher education / website and student-services AI · United States (South Dakota)
- Publisher
- The Augie Mirror (student newspaper)
- Original publication
- September 25, 2026
- Source retrieved
- 2026-10-05
- Event date
- 2026-08-15
What happened
Over several weeks after the Aug. 15 launch, Mirror reporters tested 'Ask Augustana', a custom Google Gemini assistant built with Zion Cloud Solutions to unify the university's websites. Leaders said the grounded agent 'doesn't hallucinate.' Testing found it gave a student the opposite of Financial Aid's guidance on a meal-plan scholarship question, described a DEI office dissolved in 2025 and a discontinued major as current, repeated a former professor as department chair when challenged, and said campus safety officers are unarmed (faithful to an outdated page). Rephrased prompts produced near-complete disclosure of its rules and tool name, and it answered off-scope math and Klingon requests despite an English-only patch.
Why it matters
Direct campus-operations signal: public-facing admissions, aid and student-life chatbots inherit every stale page on institutional sites, so content governance and web lifecycle become AI operating prerequisites. The university plans an advising agent next, raising the stakes.
Evidence and measured results
Opened the Mirror article (Sept. 25, 2026) describing specific test prompts, screenshots and responses, plus on-record statements from the VP for Innovation and the Zion Cloud developer.
Limitations and uncertainty
Student-newspaper testing is non-systematic, with no stated sample size or error rate; some answers changed during testing as developers patched issues. The vendor and university dispute that the model itself hallucinates.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-04; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Public-facing campus chatbot gives confident but outdated or contradictory answers on aid, programs and safety because it retrieves from stale institutional pages.
- Stakeholders
- VP for innovation / CIO
- Web and marketing content owners
- Financial aid director
- Registrar and academic affairs
- Campus safety communications
- Discovery questions
- Who owns each domain and page the chatbot indexes, and how are retired pages removed?
- How are wrong answers reported and how fast are they corrected?
- Has the bot been tested on high-stakes aid and safety questions against current policy?
- Is the catalog on a separate domain included in retrieval?
- Value hypothesis
- Interpretation: a content-readiness audit and red-team test could reduce high-stakes misinformation before the institution expands to advising agents; size of benefit is unknown.
- Potential engagement
- A bounded 4-6 week content-estate audit and chatbot accuracy and prompt-security test, ending with a prioritized remediation list and regression question set.
- Unsupported claims
- The evidence does not show any measured error rate or call deflection.
- It cannot support claims that grounding eliminates hallucination or wrong answers.
- It does not show student harm beyond individual reported cases.
Pre-sales engineering
Role takeaway
- Fit
- Retrieval-grounded assistants over multi-domain .edu web estates used for admissions, aid and student-life questions.
- Architecture and integration
- Add freshness and status metadata to indexed pages, exclude archived content, include catalog domains, enforce citation display, and log answers for review.
- Prerequisites
- Inventory of indexed domains and owners
- Page retirement and archive policy
- Current authoritative answers for high-stakes questions
- Answer logging with privacy review
- Constraints
- Separate vendor-hosted domains may not be crawled
- Public bot cannot verify student identity or individual eligibility
- Vendor model and prompt layer controlled by third party
- Security
- Test for system-prompt and tool-name disclosure via rephrasing, scope bypass, and instruction override; treat future advising-agent data access as a separate, higher-risk authorization.
- Proof of value
- Score a fixed set of 50+ aid, program and safety questions against owner-approved answers
- Rerun the set after content changes to detect regressions
- Run a prompt-disclosure and scope-bypass test script and record pass/fail
Delivery
Role takeaway
- Work
- Stand up content governance, a high-stakes regression test suite and an error-correction workflow for the public chatbot before adding advising capabilities.
- Dependencies
- Web content owners across offices
- Vendor (Zion Cloud-type) change process
- Financial aid and registrar policy owners
- Ownership
- A named service owner in the innovation or IT office, with each content domain owned by its operating office.
- Skills and adoption
- Content owners need training on how their pages feed the bot and how to retire or flag outdated material; staff need a reporting path for bad answers.
- Governance checkpoints
- Content-owner sign-off on high-stakes answers before launch of new scopes
- Prompt-security test pass before any expansion
- Advising-agent data access review before student-record integration
- Acceptance criteria
- Proposed: at least 95% correct on the owner-approved high-stakes question set
- Proposed: reported errors corrected within 5 business days
- Proposed: no system-prompt disclosure in the standard test script
- Risks
- Students acting on contradictory aid guidance
- Public reputational damage from student testing
- Scope creep into advising before content is fixed
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Retrieval-grounded design over multiple domains (augie.edu, library, athletics) did not prevent stale-source answers or missed content on a separate catalog domain (smartcatalogiq); source freshness metadata and archived-page exclusion must be engineered, not assumed.
Governance
Who approves, reviews and stays accountable for outcomes?
Leadership framed errors as 'garbage in, garbage out', which places accountability on web content owners; institutions need named owners for each indexed source, takedown of retired pages, and a correction loop before expanding to advising.
Security and privacy
What data, permissions and controls need testing?
Prompt rephrasing extracted system rules and internal tool names; scope and language restrictions were bypassed. No sensitive data exposure was reported, but the same weaknesses would matter for an advising agent that touches student records.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
The bot complied with an 'accessibility accommodation' request to speak like a pirate, showing accommodation framing can be used to override behavior; staff still carry correction work, so the tool shifts rather than removes front-office labor.
Procurement
What should contracts, pricing and exit terms secure?
Contracts with AI integrators should require accuracy testing on institution-defined question sets, prompt-security testing, content-coverage disclosure and a correction SLA, rather than accepting 'grounded, does not hallucinate' claims.
Operating model
Which teams own the service once it runs?
Moves website stewardship into the AI service chain: content retirement and ownership become operational controls for any student-facing agent.
Publication history
- 2026-10-04Campus Operations · Issue 274 resources
Stable resource ID: augie-mirror-ask-augustana-chatbot-test-20260925