From the Red Hat edition of October 29, 2026
Evaluating and Preventing Security Smells in AI-Generated Ansible Code (Preprint, not peer reviewed)
University of Waikato, School of Computing and Mathematical Sciences · Academic research: AI-generated infrastructure-as-code security · Hamilton, New Zealand (global applicability)
- Publisher
- arXiv
- Original publication
- arXiv preprint 2608.24962v1, Aug. 25, 2026
- Source retrieved
- 2026-10-30
- Event date
- 2026-08-25
What happened
Preprint, not peer reviewed. The authors evaluated 16 AI models generating Ansible roles for Apache Tomcat 10 and MongoDB 7 and analyzed 278 roles (including human-written roles from Ansible Galaxy) against CIS Level 1 benchmarks, using deployment testing with CIS-CAT Pro. Without security guidance, all 16 models produced code with security smells, such as hardcoded credentials, and underperformed human roles. With structured prompts embedding Ansible best practices and CIS controls (an extended CO-STAR framework), 4 of 16 models produced compliant code. The best reached 95 to 100% CIS compliance against 23 to 43% for human-written roles. The other 12 failed to follow multi-constraint instructions.
Why it matters
State and local IT teams and university system administrators are adopting AI assistants for Ansible automation, including Red Hat Ansible Lightspeed. The study suggests unguided AI-generated playbooks are likely to miss CIS hardening controls that many state security standards reference. Prompt-embedded standards and post-deployment compliance scans should be required, whichever assistant is used.
Evidence and measured results
Read the arXiv abstract page and the PDF. Two applications only (Tomcat, MongoDB), CIS Level 1 controls, zero-shot baseline prompts, deployment validation with CIS-CAT Pro. Red Hat Ansible Lightspeed and IBM Granite code models were not among the models tested, so results do not directly measure Red Hat's product.
Limitations and uncertainty
Preprint, not peer reviewed. Limited to two applications and CIS Level 1. Red Hat's assistant was not tested. Model versions will age quickly.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-30; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- IT teams want faster automation with AI assistants but cannot accept playbooks that miss mandatory hardening controls.
- Stakeholders
- Infrastructure automation lead
- CISO / compliance
- System administrators
- Change advisory board
- Discovery questions
- Which hardening baselines (CIS, state standards) must automation meet?
- Are AI assistants already used to write Ansible?
- Do you scan deployed systems for compliance after automation runs?
- Value hypothesis
- Combining an AI assistant with standards-embedded prompts and compliance scanning could speed automation while keeping baseline compliance.
- Potential engagement
- Governed AI-assisted automation pilot with compliance scanning on one application baseline.
- Unsupported claims
- That Ansible Lightspeed output is secure or insecure (not tested)
- That results generalize beyond Tomcat and MongoDB
- Productivity gains
Pre-sales engineering
Role takeaway
- Fit
- Applies to any AI-assisted Ansible content creation, including Lightspeed, in regulated environments.
- Architecture and integration
- Prompt templates with embedded controls, ansible-lint and policy checks in CI, vault-based secrets, post-deployment compliance scanning.
- Prerequisites
- Documented hardening baseline
- CI pipeline for Ansible content
- Compliance scanning tool
- Constraints
- Preprint evidence only
- Two applications studied
- Model capability varies widely
- Security
- Block hardcoded credentials, enforce vault use and verify hardening on deployed systems before promotion.
- Proof of value
- Generate roles for one application with and without standards-embedded prompts and compare compliance scan results
- Measure reviewer time per AI-assisted change
- Confirm zero hardcoded secrets reach the main branch
Delivery
Role takeaway
- Work
- Integrate AI-generated Ansible into a governed pipeline with linting, scanning and human approval.
- Dependencies
- CI tooling
- Compliance scanner licensing
- Administrator training
- Ownership
- Automation team owns the pipeline; security owns baselines and scan policy.
- Skills and adoption
- Train administrators in standards-aware prompting and in reviewing generated automation.
- Governance checkpoints
- Compliance scan before production promotion
- Human approval for AI-assisted changes
- Quarterly review of scan pass rates
- Acceptance criteria
- Proposed: AI-assisted roles pass the agency baseline scan before production
- Proposed: no hardcoded credentials detected in CI
- Proposed: documented comparison of AI-assisted and human pass rates
- Risks
- Over-trust in syntactically valid but insecure code
- Baseline drift between prompts and policy
- Reviewer fatigue
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
AI-assisted automation pipelines should include standards-aware prompt templates, linting and post-deployment compliance scanning, not just syntax checks.
Governance
Who approves, reviews and stays accountable for outcomes?
Agencies can require that AI-generated automation be validated against CIS or state baselines before merge and deployment.
Security and privacy
What data, permissions and controls need testing?
Unguided outputs included security smells such as hardcoded credentials and missing hardening. Successful guided outputs used Ansible Vault and proper file permissions.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Administrators need training to write standards-aware prompts and to review generated automation critically.
Procurement
What should contracts, pricing and exit terms secure?
Evaluations of AI automation assistants should test output against the buyer's hardening baselines, since model capability to follow multi-constraint instructions varied widely.
Operating model
Which teams own the service once it runs?
Treat AI-generated automation as untrusted code entering the same review and compliance pipeline as human contributions.
Publication history
- 2026-10-29Red Hat · Issue 018 resources
Stable resource ID: waikato-arxiv-security-smells-ai-generated-ansible-20260825