{"resourceId":"nizamani-vt-llm-use-across-cs-subfields-20261001","versions":[{"version":"external-a1a8cc3036f7e13952d73f1a33a88c5b51110933fea6d495d96cfee6736750a7","resource":{"id":"nizamani-vt-llm-use-across-cs-subfields-20261001","title":"VT cross-subfield study: LLM adoption swings from 89.6% (algorithms) to 15.2% (software engineering) under one literacy intervention","organization":"Virginia Tech Department of Computer Science (Nizamani, Lee, Donekal Chandrashekar, Ellis, Ramakrishnan)","sector":"Higher education computing teaching / AI literacy and student GenAI practices","geography":"United States (Virginia Tech; undergraduate post-CS1 problem-solving course)","publishedAt":"Submitted to arXiv 1 October 2026 (cs.CY); accepted IEEE FIE 2026","publicationDate":"2026-10-01","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (course reflection study)","sourceUrl":"https://arxiv.org/pdf/2610.01158","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents","operating-model","accessibility-workforce"],"finding":"Among 211 undergraduates completing reflections across seven CS subfield assignments under a shared demystify–use–reflect intervention, LLM adoption varied from Algorithms 89.6% and Web Development 80.6% down to Software Engineering 15.2%, with Networking/Databases moderate (~58.3%) and HCI/ML lower (~40–43%). Students predominantly cast LLMs as Assistants rather than Experts. Among users, verification was nearly universal (93%–100%), typically multi-strategy; testing was highest in Algorithms (87.8%) and lowest in HCI (27.5%), while web search rose in open-ended tasks.","sledRelevance":"Shows one campus AI-literacy policy does not produce uniform GenAI practice: assignment structure drives adoption and verification mix. Complements yesterday's enjoyment-vs-actionable-feedback and engagement-gate findings by warning student-success teams not to treat 'students use Copilot' as a single behavior.","evidence":"Post-assignment reflection surveys; Kruskal–Wallis on prompt counts; χ² on testing rates. Self-report not telemetry; single course; preprint.","architectureImplications":"Instrument GenAI use and verification by assignment type in LMS analytics; avoid one-size telemetry dashboards that average across subfields.","governanceImplications":"Write AI-literacy policies that vary expectations by task type (testable coding vs open design) rather than a single allowed/banned rule.","securityPrivacyImplications":"Reflection responses and prompt-count data are education records; avoid joining to misconduct systems without policy review.","caveats":"Self-reported; assignment design confounded with subfield; no learning RCT; single R1 institution; preprint.","streamIds":["student-success"],"roles":{"sales":"For campus AI-literacy buyers assuming uniform Copilot habits, Virginia Tech's N=211 cross-subfield study shows adoption swinging from 89.6% in algorithms to 15.2% in software engineering under one reflective intervention, with nearly universal multi-strategy verification among users. Sell assignment-aware literacy analytics and coaching—not a single campus GenAI mandate. Bound claims to self-reported course reflections and refuse guaranteed learning gains.","engineering":"Engineering should capture prompt-intensity and verification-method signals per assignment type, and support faculty-configured literacy reflections. Do not average GenAI usage into one campus KPI. Constraints include self-report evidence and single-course external validity. Proof-of-value is actionable subfield differences that change coaching—not raw adoption upticks. Keep reflection data purpose-limited.","delivery":"Delivery should run demystify–use–reflect modules, collect post-assignment reflections, and brief CS faculty on subfield adoption gaps. Ownership sits with the course lead and teaching center. Acceptance means published internal adoption/verification breakdowns by assignment family. Risks include treating self-report as telemetry and over-generalizing from one R1 course."},"roleDetails":{"sales":{"customerProblem":"Institutions set one GenAI literacy rule while student use and verification differ radically by assignment type.","stakeholders":["CS faculty","teaching centers","academic technology","student success","academic integrity offices"],"discoveryQuestions":["Do we measure GenAI adoption by assignment type?","What verification strategies do students report?","Is our literacy intervention reflective or ban-only?"],"valueHypothesis":"Assignment-aware literacy analytics can target coaching where adoption is high and verification weak, without claiming grade guarantees.","potentialEngagement":"One-term reflection instrumentation pilot across a multi-module CS course with faculty readout.","unsupportedClaims":["Causal learning gains","Telemetry-grade accuracy from surveys","Uniform campus adoption"]},"engineering":{"fit":"Undergraduate CS courses seeking responsible GenAI literacy measurement across diverse assignment types.","architectureIntegration":"LMS reflection forms → per-assignment analytics warehouse → faculty dashboards by subfield/activity family.","prerequisites":["LMS survey tooling","roster linkage","IRB/FERPA review"],"constraints":["self-report bias","single-site evidence","assignment confound"],"security":"Reflection and roster joins as education records; minimize retention; no automatic misconduct routing.","proofOfValue":["Stable per-assignment adoption reports","Verification mix visible to faculty","Coaching changes documented"]},"delivery":{"work":"Deploy reflection prompts, clean response data, produce subfield adoption/verification briefings, iterate coaching.","dependencies":["course instructor","teaching center","IR/assessment"],"ownership":"Course instructor with teaching-center analytics support.","skillsAdoption":"Faculty interpret subfield differences; students practice verification strategies matched to task type.","governanceCheckpoints":["FERPA/IRB signoff","mid-term data quality check","end-term faculty readout"],"acceptanceCriteria":["Adoption by assignment published internally","Verification strategies summarized","No ban-only policy assumed"],"risks":["Self-report inflation","Over-policing high-adoption courses","Ignoring open-ended task needs"]}},"retrievedAt":"2026-10-04T03:18:47Z","enrichedAt":"2026-10-04T03:21:30Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Students in open-ended tasks may need different verification scaffolds than those in unit-testable coding—uniform AI rules can under-serve or over-police subgroups.","procurementImplications":"Prefer tools that report usage and verification proxies by course activity type, with faculty-configurable literacy modules—not only seat licenses.","operatingModelImplications":"Teaching centers should review GenAI reflection data by assignment family weekly and adjust prompting/verification coaching accordingly.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2610.01158","referenceExcerpt":"Algorithms (89.6%) and Web Development (80.6%) showing the highest usage, and Software Engineering the lowest (15.2%)","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}