{"resourceId":"arxiv-sydney-ai-permissive-engineering-course-six-channel-assessment-evidence-20261026","versions":[{"version":"external-9efe1d6239561360e186f25a6f540a261e3cbd53cb12b74abf170c93ab063d35","resource":{"id":"arxiv-sydney-ai-permissive-engineering-course-six-channel-assessment-evidence-20261026","title":"Preprint: University of Sydney redesigns a postgraduate software engineering course to permit AI, assessing competence through six evidence channels and a supervised exam hurdle; first deployment with 87 students","organization":"University of Sydney, School of Electrical and Computer Engineering (author)","sector":"Higher education assessment and engineering education","geography":"Sydney, Australia","publishedAt":"arXiv preprint 2610.36413, submitted 26 October 2026","publicationDate":"2026-10-26","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (course redesign and deployment report)","sourceUrl":"https://arxiv.org/abs/2610.36413","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["developers-agents","operating-model"],"finding":"A 13-week AI-permissive postgraduate software engineering course lets students delegate substantial work to AI while keeping responsibility for planning, verification, review, integration and explanation. Assessment draws on six channels (product, process/origin, verification, explanation, reflection and individual assurance) linked by a collect, cross-check and escalate logic, with targeted oral follow-up where evidence is inconsistent. A supervised final exam worth 30% carries a hurdle of at least 30% of exam marks. By week 11 the deployment involved 87 students in 17 teams, 957 individual weekly Git repositories and 34 group repositories. Lessons: AI changes where rubric criteria carry evidential weight, channels need deliberate cross-checking, realised evidential strength may differ from design, and automation scales evidence collection more readily than interpretation. The author reports substantial human effort, concentrated in the coordinator role, and no outcome analysis.","sledRelevance":"Offers a concrete design for assessing competence when AI use is allowed, complementing secured-assessment approaches with process and verification evidence.","evidence":"Design description and deployment counts at week 11; no comparison group or marks.","architectureImplications":"Git-based workflows, automated repository provisioning, autograders and coordinator-built repository-analysis tools feed assessment evidence.","governanceImplications":"Separates claims about AI-assisted performance from individually attributable competence; supervised exam hurdle retained.","securityPrivacyImplications":"AI-use journals and repository traces are assessed student records; handling not detailed.","caveats":"Single offering; technically favourable setting; author notes AI-assisted students may produce correlated evidence across channels and that assessed traces can become compliance-oriented.","streamIds":["student-success"],"roles":{"sales":"Sales framing: this Sydney preprint shows one way to allow substantial AI use while still assessing individual competence: multiple evidence channels, targeted oral follow-up and a supervised exam hurdle. It suits conversations with engineering and computing deans redesigning assessment. A bounded engagement could map one course's assessments to evidence channels and estimate staffing. Do not claim the redesign improves learning or validity; there was no comparison group and no marks were analysed.","engineering":"Engineering framing: the enabling stack is Git-based workflows, automated repository provisioning, autograders and tools that summarise commits, reviews and contributions for assessors. Check whether an institution can provision hundreds of repositories, retain AI-use journals securely, and generate assessor summaries reliably. Proof of value should test whether repository analytics actually help assessors detect inconsistent evidence, since the author notes automation scales collection faster than interpretation.","delivery":"Delivery framing: delivery is course redesign plus workload planning. Work includes defining which evidence channels each rubric criterion relies on, training tutors in cross-checking and escalation, scheduling oral follow-up, and keeping a supervised hurdle for individual assurance. Budget for coordinator effort, which the author describes as substantial, and plan an outcome evaluation with marks and student feedback before scaling to larger or less software-intensive courses."},"roleDetails":{"sales":{"customerProblem":"Programmes that allow AI use cannot rely on artefact quality as evidence of student competence.","stakeholders":["Engineering and computing deans","Course coordinators","Assessment and quality office","Tutors"],"discoveryQuestions":["Which assessment criteria in your AI-permissive courses depend on artefacts AI can produce?","Do you keep a supervised component with a hurdle?","How much coordinator time could evidence cross-checking take?"],"valueHypothesis":"Interpretation: multi-channel evidence with selective escalation could keep competence claims credible while permitting AI.","potentialEngagement":"Assessment evidence mapping and workload estimate for one AI-permissive course.","unsupportedClaims":["Improved learning or competence","Validated measurement of competence","Low staffing cost"]},"engineering":{"fit":"Software-intensive courses with Git workflows and autograding infrastructure.","architectureIntegration":"Automated repository provisioning, autograders, AI-use journal processing and assessor-facing repository analytics.","prerequisites":["Git hosting at course scale","Autograder infrastructure","Secure storage for AI-use journals"],"constraints":["Single postgraduate course","Technically favourable setting","Correlated evidence across channels"],"security":"Restrict access to journals and repository analytics to assessors; set retention for assessment traces.","proofOfValue":["Assessor use of analytics to flag inconsistent evidence","Escalation rate and outcome of oral follow-up","Coordinator hours tracked"]},"delivery":{"work":"Map criteria to evidence channels, train tutors in cross-checking, run oral follow-up and keep a supervised hurdle.","dependencies":["Coordinator capacity","Tutor hours","Repository automation"],"ownership":"Course coordinator with the school teaching committee.","skillsAdoption":"Tutor training on evidence cross-checking and escalation; student guidance on verification responsibilities.","governanceCheckpoints":["School approval of assessment design","Mid-term review of escalation cases","End-of-course outcome evaluation"],"acceptanceCriteria":["Proposed: each rubric criterion linked to named evidence channels","Proposed: coordinator and tutor hours recorded","Proposed: outcome evaluation with marks completed before scaling"],"risks":["Coordinator overload","Compliance-oriented reflections","Evidence that looks independent but is AI-correlated"]}},"retrievedAt":"2026-10-29T03:19:50Z","enrichedAt":"2026-10-29T03:24:23Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Substantial coordinator and tutor workload despite automation; tutors had set weekly hours.","procurementImplications":"None stated.","operatingModelImplications":"Interpretation: AI-permissive assessment shifts staff effort from marking artefacts to cross-checking evidence and running follow-up.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.36413","referenceExcerpt":"By the end of Week 11, the deployment involved 87 students organised into 17 teams","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}