{"resourceId":"waikato-arxiv-security-smells-ai-generated-ansible-20260825","versions":[{"version":"external-2b5695e6d7c6d3ec839d6cac5bd1318f71861d3cfc0cc0cb97f6aebd3d038535","resource":{"id":"waikato-arxiv-security-smells-ai-generated-ansible-20260825","title":"Evaluating and Preventing Security Smells in AI-Generated Ansible Code (Preprint, not peer reviewed)","organization":"University of Waikato, School of Computing and Mathematical Sciences","sector":"Academic research: AI-generated infrastructure-as-code security","geography":"Hamilton, New Zealand (global applicability)","publishedAt":"arXiv preprint 2608.24962v1, Aug. 25, 2026","publicationDate":"2026-08-25","eventDate":"2026-08-25","sourceName":"arXiv","sourceLabel":"arXiv:2608.24962, Evaluating and Preventing Security Smells in AI-Generated Ansible Code","sourceUrl":"https://arxiv.org/abs/2608.24962","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","data-security","governance-procurement"],"finding":"Preprint, not peer reviewed. The authors evaluated 16 AI models generating Ansible roles for Apache Tomcat 10 and MongoDB 7 and analyzed 278 roles (including human-written roles from Ansible Galaxy) against CIS Level 1 benchmarks, using deployment testing with CIS-CAT Pro. Without security guidance, all 16 models produced code with security smells, such as hardcoded credentials, and underperformed human roles. With structured prompts embedding Ansible best practices and CIS controls (an extended CO-STAR framework), 4 of 16 models produced compliant code. The best reached 95 to 100% CIS compliance against 23 to 43% for human-written roles. The other 12 failed to follow multi-constraint instructions.","sledRelevance":"State and local IT teams and university system administrators are adopting AI assistants for Ansible automation, including Red Hat Ansible Lightspeed. The study suggests unguided AI-generated playbooks are likely to miss CIS hardening controls that many state security standards reference. Prompt-embedded standards and post-deployment compliance scans should be required, whichever assistant is used.","evidence":"Read the arXiv abstract page and the PDF. Two applications only (Tomcat, MongoDB), CIS Level 1 controls, zero-shot baseline prompts, deployment validation with CIS-CAT Pro. Red Hat Ansible Lightspeed and IBM Granite code models were not among the models tested, so results do not directly measure Red Hat's product.","architectureImplications":"AI-assisted automation pipelines should include standards-aware prompt templates, linting and post-deployment compliance scanning, not just syntax checks.","governanceImplications":"Agencies can require that AI-generated automation be validated against CIS or state baselines before merge and deployment.","securityPrivacyImplications":"Unguided outputs included security smells such as hardcoded credentials and missing hardening. Successful guided outputs used Ansible Vault and proper file permissions.","caveats":"Preprint, not peer reviewed. Limited to two applications and CIS Level 1. Red Hat's assistant was not tested. Model versions will age quickly.","streamIds":["red-hat"],"roles":{"sales":"Sales framing: use this to sell governed automation, not to promise secure-by-default AI playbooks. The preprint found every tested model produced insecure Ansible without guidance, while structured, standards-embedded prompting made a few models outperform human roles on CIS compliance. For SLED customers considering Ansible Lightspeed, pair it with Ansible Automation Platform controls, linting and compliance scanning. Do not cite this as evidence about Lightspeed itself; it was not tested.","engineering":"Engineering framing: build guardrails into the automation pipeline. Use prompt templates that embed the agency's CIS or state baseline controls, run ansible-lint and policy checks on generated content, and verify deployed systems with a compliance scanner before promotion. Keep credentials in a vault and block hardcoded secrets in CI. If evaluating Lightspeed or any assistant, test it against the customer's own baseline roles rather than relying on general benchmarks.","delivery":"Delivery framing: add AI-generated automation to the existing change pipeline with mandatory review, linting and compliance scanning. Train administrators on standards-aware prompting and critical review of generated roles. Track compliance scan pass rates for AI-assisted versus human-written changes as an adoption metric, and keep human approval for production promotion until results are proven in the customer's environment. Start with one well-understood application baseline."},"roleDetails":{"sales":{"customerProblem":"IT teams want faster automation with AI assistants but cannot accept playbooks that miss mandatory hardening controls.","stakeholders":["Infrastructure automation lead","CISO / compliance","System administrators","Change advisory board"],"discoveryQuestions":["Which hardening baselines (CIS, state standards) must automation meet?","Are AI assistants already used to write Ansible?","Do you scan deployed systems for compliance after automation runs?"],"valueHypothesis":"Combining an AI assistant with standards-embedded prompts and compliance scanning could speed automation while keeping baseline compliance.","potentialEngagement":"Governed AI-assisted automation pilot with compliance scanning on one application baseline.","unsupportedClaims":["That Ansible Lightspeed output is secure or insecure (not tested)","That results generalize beyond Tomcat and MongoDB","Productivity gains"]},"engineering":{"fit":"Applies to any AI-assisted Ansible content creation, including Lightspeed, in regulated environments.","architectureIntegration":"Prompt templates with embedded controls, ansible-lint and policy checks in CI, vault-based secrets, post-deployment compliance scanning.","prerequisites":["Documented hardening baseline","CI pipeline for Ansible content","Compliance scanning tool"],"constraints":["Preprint evidence only","Two applications studied","Model capability varies widely"],"security":"Block hardcoded credentials, enforce vault use and verify hardening on deployed systems before promotion.","proofOfValue":["Generate roles for one application with and without standards-embedded prompts and compare compliance scan results","Measure reviewer time per AI-assisted change","Confirm zero hardcoded secrets reach the main branch"]},"delivery":{"work":"Integrate AI-generated Ansible into a governed pipeline with linting, scanning and human approval.","dependencies":["CI tooling","Compliance scanner licensing","Administrator training"],"ownership":"Automation team owns the pipeline; security owns baselines and scan policy.","skillsAdoption":"Train administrators in standards-aware prompting and in reviewing generated automation.","governanceCheckpoints":["Compliance scan before production promotion","Human approval for AI-assisted changes","Quarterly review of scan pass rates"],"acceptanceCriteria":["Proposed: AI-assisted roles pass the agency baseline scan before production","Proposed: no hardcoded credentials detected in CI","Proposed: documented comparison of AI-assisted and human pass rates"],"risks":["Over-trust in syntactically valid but insecure code","Baseline drift between prompts and policy","Reviewer fatigue"]}},"retrievedAt":"2026-10-30T03:09:23Z","enrichedAt":"2026-10-30T03:21:15Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Administrators need training to write standards-aware prompts and to review generated automation critically.","procurementImplications":"Evaluations of AI automation assistants should test output against the buyer's hardening baselines, since model capability to follow multi-constraint instructions varied widely.","operatingModelImplications":"Treat AI-generated automation as untrusted code entering the same review and compliance pipeline as human contributions.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2608.24962","referenceExcerpt":"Without security guidance, all 16 AI models produced code containing security smells, resulting in vulnerable infrastructure that fails compliance verification","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}