{"resourceId":"fabrega-epistemic-ownership-studychat-20261002","versions":[{"version":"external-6b418884b46da06fc6084f6a7276ec0eceb910edb1052b2b80cc12e448955bc9","resource":{"id":"fabrega-epistemic-ownership-studychat-20261002","title":"Same score, different process: in 150 student–AI conversations, assignment grades did not reveal how cognitive work was shared","organization":"Jorge Fábrega (CICS, Universidad del Desarrollo, Chile), analyzing the StudyChat dataset (McNichols, Ikram, Lan)","sector":"Higher education assessment / GenAI-mediated coursework in AI and computer science","geography":"Author in Chile; data from U.S. university StudyChat corpus (Fall 2024–Spring 2025)","publishedAt":"Submitted to arXiv 2 October 2026 (cs.CY)","publicationDate":"2026-10-02","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (secondary analysis of course chat logs)","sourceUrl":"https://arxiv.org/pdf/2610.02731","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","developers-agents","governance-procurement","operating-model"],"finding":"Using StudyChat (2,214 conversations from 203 students on 7 AI/CS assignments), the author coded 150 sampled conversations for observable Direction, Integration, and Evaluation. Direction appeared in 97.3%, Integration in 37.3%, and Evaluation in 46.0%, rising to 90.5% each in conversations with 15+ student turns. Scores linked for 134 conversations (median 0.980; 41.0% at 1.00) showed no stable performance difference between conversations with all three dimensions observed and those missing one. Matched pairs with identical scores showed different participation patterns.","sledRelevance":"Fresh evidence for campuses relying on grades or completion as proof that GenAI-assisted work reflects learning. It complements the UMD RCT and yesterday's oversight framing: product scores alone can't show whether students directed and judged the AI's work.","evidence":"Descriptive secondary analysis; LLM-assisted coding (GPT-based, schema-constrained) with manual supervision of a random subsample; proportional stratified sample of 150; score linkage for 134; no causal design. Ceiling effects in scores (41% perfect).","architectureImplications":"Process evidence requires retaining structured chat logs linked to assignments and students, with coding pipelines that are auditable; LLM-based coding of student transcripts needs its own validation.","governanceImplications":"Assessment policies for GenAI-permitted work should consider process evidence or follow-up elicitation (answerability), not just product quality; any process monitoring needs transparency to students.","securityPrivacyImplications":"Analyzing student–AI transcripts is sensitive; use de-identified data, purpose limits, and student notice; sending transcripts to an external LLM for coding is a data-transfer decision.","caveats":"Small sample; single CS/AI course context with fast executable feedback that makes process unusually visible; LLM-coded with partial manual check; score ceiling limits contrasts; author notes the sample cannot support population-level claims; answerability not measured.","streamIds":["student-success"],"roles":{"sales":"The framing point is that grades no longer prove learning when GenAI is in the loop. In this study, near-identical scores came with very different levels of student direction and judgment. Use it with academic leaders worried about assessment validity: the conversation is about adding process evidence or brief defense steps, not surveillance. Keep claims modest, since this is a 150-conversation descriptive study in one CS course context.","engineering":"For engineering, the point is that process evidence depends on log design. Store student–AI turns with assignment and interaction-order identifiers, link them to grades under strict access control, and make any automated coding auditable, since this study used an LLM coder with partial human review. Avoid sending identifiable transcripts to external models without an agreement. Build analytics that report observed direction, integration, and evaluation as descriptive signals, never as a score.","delivery":"Delivery should pilot one GenAI-permitted assignment that adds a short answerability check, such as a reflection or oral defense, and compare it with chat-log indicators. Give students notice and keep analytics formative. Train faculty to read absence of evidence carefully, since short chats leave few traces. Use the pilot to decide whether process evidence is worth the faculty time before expanding it to other courses."},"roleDetails":{"sales":{"customerProblem":"Faculty can't tell from a polished GenAI-assisted submission how much of the thinking the student did.","stakeholders":["Provost / assessment office","Faculty senate assessment committee","Center for teaching and learning","Academic integrity office"],"discoveryQuestions":["Which courses permit GenAI on graded work?","Do you retain student–AI chat logs tied to assignments?","Is there appetite for brief oral or written defense steps?"],"valueHypothesis":"Adding process evidence or answerability checks can make GenAI-permitted assessment more valid without banning AI.","potentialEngagement":"Assessment-design pilot in one course pairing chat-log indicators with an answerability step.","unsupportedClaims":["That process indicators predict learning","Automated integrity detection","Generalization beyond CS tasks"]},"engineering":{"fit":"Courses with institution-hosted GenAI assistants whose logs can be linked to assignments.","architectureIntegration":"Chat platform logs → de-identified store → auditable coding pipeline → instructor dashboard of descriptive indicators.","prerequisites":["Institution-controlled chat logs","Data governance approval","Validation set for automated coding"],"constraints":["External-LLM coding of transcripts","Short conversations yield little evidence"],"security":"De-identify transcripts, restrict linkage to grades, and document any external model processing.","proofOfValue":["Inter-rater agreement between automated and human coding","Indicator stability across assignments","Faculty usefulness ratings"]},"delivery":{"work":"Design an answerability step, configure logging, run one-term pilot, and review indicators with faculty.","dependencies":["Faculty willing to redesign one assessment","IRB or data-governance sign-off"],"ownership":"Center for teaching and learning with the course instructor.","skillsAdoption":"Students learn to direct and evaluate AI output explicitly; faculty learn to read process evidence formatively.","governanceCheckpoints":["Student notice and consent language","Coding validation before use","No use of indicators in grading during pilot"],"acceptanceCriteria":["Defined human–automated coding agreement threshold","Faculty find indicators useful in a post-pilot survey","Answerability step completed by most students"],"risks":["Perceived surveillance","Penalizing short but legitimate use","Added faculty workload"]}},"retrievedAt":"2026-10-05T03:17:35Z","enrichedAt":"2026-10-05T03:21:28Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Process-based assessment adds faculty review workload; short conversations offer few traces, so absence of evidence should not penalize students.","procurementImplications":"Ask tutor and chatbot vendors whether logs are exportable per assignment and student, and whether analytics distinguish student direction and evaluation from AI output.","operatingModelImplications":"Pair GenAI-permitted assignments with an answerability step such as an oral or written defense, and treat chat-log analytics as formative signals rather than grading inputs.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2610.02731","referenceExcerpt":"Similar assessed performance can accompany different observable patterns of Direction, Integration, and Evaluation.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}