From the AWS edition of November 10, 2026
AWS publishes an agent evaluation playbook for AgentCore, adding managed DeepEval and AutoEval evaluators alongside built-in ones
Amazon Web Services · Cross-SLED agent platforms / AI assurance · Not region-specific in source
- Publisher
- AWS Artificial Intelligence Blog
- Original publication
- 2026-11-10
- Source retrieved
- 2026-11-11
- Event date
- 2026-11-10
What happened
AWS's November 10, 2026 post describes AgentCore Evaluations with three tiers: AWS-authored built-in evaluators (GoalSuccessRate, Helpfulness and 13 more), managed third-party evaluators from the open-source DeepEval and AutoEval libraries, and custom LLM-as-a-judge evaluators including multi-dimension scoring. It recommends defining success in business terms, mapping behaviors to trace- or session-level evaluators, and running on-demand, online (sampled live traffic) and batch (golden dataset) modes. It warns that a high completion rate can hide failures such as PII leakage, and that ground-truth evaluators run only in batch.
Why it matters
Agencies deploying resident- or student-facing agents increasingly need documented evaluation for AI governance reviews. This gives AWS customers a managed way to score agents on PII leakage, goal completion and policy compliance, though results depend on judge models and rubrics the agency must validate.
Evidence and measured results
Vendor guidance. No customer results, accuracy figures for the evaluators, or Region and GovCloud availability are stated.
Limitations and uncertainty
Vendor guidance; evaluator reliability is asserted ('benchmarked for reliability') without published data. GovCloud availability not stated.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-11; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies run agents without measured evidence of task success, policy compliance or PII leakage.
- Stakeholders
- AI governance lead
- Privacy officer
- Agent product owner
- Discovery questions
- What evidence does your AI governance process require for an agent?
- Do you have a labeled test set of real requests?
- Which failure would be most harmful?
- Value hypothesis
- Managed evaluation can make agent quality visible and gate releases; evaluator accuracy must be validated.
- Potential engagement
- Set up an evaluation baseline for one production agent.
- Unsupported claims
- Evaluator accuracy
- GovCloud availability
- Any improvement in agent outcomes
Pre-sales engineering
Role takeaway
- Fit
- AgentCore-based agents in Regions where Evaluations is offered.
- Architecture and integration
- Evaluators on AgentCore traces; on-demand, online and batch modes.
- Prerequisites
- AgentCore runtime tracing
- Golden dataset
- Agreed thresholds
- Constraints
- LLM-judge error
- Ground-truth evaluators batch-only
- Region availability not stated
- Security
- Include PII leakage and security evaluators; restrict access to evaluation traces containing personal data.
- Proof of value
- Evaluator vs human agreement on a sample
- Regression detection on seeded failures
Delivery
Role takeaway
- Work
- Establish a per-release evaluation gate for one agent.
- Dependencies
- Dataset
- Reviewers
- Thresholds
- Ownership
- Agent product owner; governance board approves thresholds.
- Skills and adoption
- Teams learn rubric design and judge validation.
- Governance checkpoints
- Threshold approval
- Per-criterion release gate
- Quarterly dataset refresh
- Acceptance criteria
- Proposed: agreed evaluator-human agreement level
- Proposed: zero releases with PII-leakage regressions
- Risks
- Over-trust in judges
- Stale golden datasets
- Trace data exposure
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Evaluators attach to AgentCore runtime traces; online evaluation samples live traffic; batch evaluation runs against golden datasets for release gating.
Governance
Who approves, reviews and stays accountable for outcomes?
Multi-dimension scoring lets agencies gate releases on each criterion (for example compliance separately from tone), which suits AI policy reviews.
Security and privacy
What data, permissions and controls need testing?
PII leakage evaluators (DeepEval) and a security evaluator for malicious output are listed; LLM judges can themselves err.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
No accessibility evaluator is described; agencies would need custom criteria for plain language or accessibility.
Procurement
What should contracts, pricing and exit terms secure?
Pricing not stated in the post.
Operating model
Which teams own the service once it runs?
Requires owners for golden datasets and evaluation thresholds; evaluation becomes a standing operational function.
Publication history
- 2026-11-10AWS · Issue 128 resources
Stable resource ID: aws-agentcore-evaluations-playbook-deepeval-autoeval-20261110