From the Student Success edition of October 28, 2026
Preprint: University of Sydney redesigns a postgraduate software engineering course to permit AI, assessing competence through six evidence channels and a supervised exam hurdle; first deployment with 87 students
University of Sydney, School of Electrical and Computer Engineering (author) · Higher education assessment and engineering education · Sydney, Australia
- Publisher
- arXiv
- Original publication
- arXiv preprint 2610.36413, submitted 26 October 2026
- Source retrieved
- 2026-10-29
What happened
A 13-week AI-permissive postgraduate software engineering course lets students delegate substantial work to AI while keeping responsibility for planning, verification, review, integration and explanation. Assessment draws on six channels (product, process/origin, verification, explanation, reflection and individual assurance) linked by a collect, cross-check and escalate logic, with targeted oral follow-up where evidence is inconsistent. A supervised final exam worth 30% carries a hurdle of at least 30% of exam marks. By week 11 the deployment involved 87 students in 17 teams, 957 individual weekly Git repositories and 34 group repositories. Lessons: AI changes where rubric criteria carry evidential weight, channels need deliberate cross-checking, realised evidential strength may differ from design, and automation scales evidence collection more readily than interpretation. The author reports substantial human effort, concentrated in the coordinator role, and no outcome analysis.
Why it matters
Offers a concrete design for assessing competence when AI use is allowed, complementing secured-assessment approaches with process and verification evidence.
Evidence and measured results
Design description and deployment counts at week 11; no comparison group or marks.
Limitations and uncertainty
Single offering; technically favourable setting; author notes AI-assisted students may produce correlated evidence across channels and that assessed traces can become compliance-oriented.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-29; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Programmes that allow AI use cannot rely on artefact quality as evidence of student competence.
- Stakeholders
- Engineering and computing deans
- Course coordinators
- Assessment and quality office
- Tutors
- Discovery questions
- Which assessment criteria in your AI-permissive courses depend on artefacts AI can produce?
- Do you keep a supervised component with a hurdle?
- How much coordinator time could evidence cross-checking take?
- Value hypothesis
- Interpretation: multi-channel evidence with selective escalation could keep competence claims credible while permitting AI.
- Potential engagement
- Assessment evidence mapping and workload estimate for one AI-permissive course.
- Unsupported claims
- Improved learning or competence
- Validated measurement of competence
- Low staffing cost
Pre-sales engineering
Role takeaway
- Fit
- Software-intensive courses with Git workflows and autograding infrastructure.
- Architecture and integration
- Automated repository provisioning, autograders, AI-use journal processing and assessor-facing repository analytics.
- Prerequisites
- Git hosting at course scale
- Autograder infrastructure
- Secure storage for AI-use journals
- Constraints
- Single postgraduate course
- Technically favourable setting
- Correlated evidence across channels
- Security
- Restrict access to journals and repository analytics to assessors; set retention for assessment traces.
- Proof of value
- Assessor use of analytics to flag inconsistent evidence
- Escalation rate and outcome of oral follow-up
- Coordinator hours tracked
Delivery
Role takeaway
- Work
- Map criteria to evidence channels, train tutors in cross-checking, run oral follow-up and keep a supervised hurdle.
- Dependencies
- Coordinator capacity
- Tutor hours
- Repository automation
- Ownership
- Course coordinator with the school teaching committee.
- Skills and adoption
- Tutor training on evidence cross-checking and escalation; student guidance on verification responsibilities.
- Governance checkpoints
- School approval of assessment design
- Mid-term review of escalation cases
- End-of-course outcome evaluation
- Acceptance criteria
- Proposed: each rubric criterion linked to named evidence channels
- Proposed: coordinator and tutor hours recorded
- Proposed: outcome evaluation with marks completed before scaling
- Risks
- Coordinator overload
- Compliance-oriented reflections
- Evidence that looks independent but is AI-correlated
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Git-based workflows, automated repository provisioning, autograders and coordinator-built repository-analysis tools feed assessment evidence.
Governance
Who approves, reviews and stays accountable for outcomes?
Separates claims about AI-assisted performance from individually attributable competence; supervised exam hurdle retained.
Security and privacy
What data, permissions and controls need testing?
AI-use journals and repository traces are assessed student records; handling not detailed.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Substantial coordinator and tutor workload despite automation; tutors had set weekly hours.
Procurement
What should contracts, pricing and exit terms secure?
None stated.
Operating model
Which teams own the service once it runs?
AI-permissive assessment shifts staff effort from marking artefacts to cross-checking evidence and running follow-up.
Publication history
- 2026-10-28Student Success · Issue 514 resources
Stable resource ID: arxiv-sydney-ai-permissive-engineering-course-six-channel-assessment-evidence-20261026