{"resourceId":"aws-agentcore-evaluations-playbook-deepeval-autoeval-20261110","versions":[{"version":"external-68342420a81c71ffacec2dc3f16dca75a5abb78c99a08b31a6ab3dc78c9986fb","resource":{"id":"aws-agentcore-evaluations-playbook-deepeval-autoeval-20261110","title":"AWS publishes an agent evaluation playbook for AgentCore, adding managed DeepEval and AutoEval evaluators alongside built-in ones","organization":"Amazon Web Services","sector":"Cross-SLED agent platforms / AI assurance","geography":"Not region-specific in source","publishedAt":"2026-11-10","publicationDate":"2026-11-10","eventDate":"2026-11-10","sourceName":"AWS Artificial Intelligence Blog","sourceLabel":"The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations","sourceUrl":"https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["developers-agents","governance-procurement","operating-model"],"finding":"AWS's November 10, 2026 post describes AgentCore Evaluations with three tiers: AWS-authored built-in evaluators (GoalSuccessRate, Helpfulness and 13 more), managed third-party evaluators from the open-source DeepEval and AutoEval libraries, and custom LLM-as-a-judge evaluators including multi-dimension scoring. It recommends defining success in business terms, mapping behaviors to trace- or session-level evaluators, and running on-demand, online (sampled live traffic) and batch (golden dataset) modes. It warns that a high completion rate can hide failures such as PII leakage, and that ground-truth evaluators run only in batch.","sledRelevance":"Agencies deploying resident- or student-facing agents increasingly need documented evaluation for AI governance reviews. This gives AWS customers a managed way to score agents on PII leakage, goal completion and policy compliance, though results depend on judge models and rubrics the agency must validate.","evidence":"Vendor guidance. No customer results, accuracy figures for the evaluators, or Region and GovCloud availability are stated.","architectureImplications":"Evaluators attach to AgentCore runtime traces; online evaluation samples live traffic; batch evaluation runs against golden datasets for release gating.","governanceImplications":"Multi-dimension scoring lets agencies gate releases on each criterion (for example compliance separately from tone), which suits AI policy reviews.","securityPrivacyImplications":"PII leakage evaluators (DeepEval) and a security evaluator for malicious output are listed; LLM judges can themselves err.","caveats":"Vendor guidance; evaluator reliability is asserted ('benchmarked for reliability') without published data. GovCloud availability not stated.","streamIds":["aws"],"roles":{"sales":"Interpretation: position evaluation as the evidence an AI governance board asks for before approving an agent. The customer problem is agents in production with no measured view of task success or data leakage. Stakeholders are the AI governance lead, privacy officer and agent product owner. Discovery should ask what evidence the agency requires to approve or renew an agent. Do not claim evaluators are accurate; their reliability must be checked against human review.","engineering":"Interpretation: fit is good for AgentCore-based agents. Engineers select two or three evaluators tied to outcomes, add PII leakage checks, and run batch evaluation on a golden dataset before each release, with online sampling in production. Prerequisites are AgentCore runtime tracing and a labeled dataset. Constraints include judge-model error, unstated GovCloud availability and ground-truth evaluators being batch-only. Proof of value compares evaluator scores with human ratings on a sample.","delivery":"Interpretation: delivery work is setting up an evaluation regime. Dependencies are a golden dataset, agreed thresholds and reviewer time. Ownership sits with the agent product owner, reporting to the AI governance board. Governance checkpoints include threshold approval and release gates per criterion. Proposed acceptance criteria: evaluator-human agreement above an agreed level on a sample, and no release with PII-leakage regressions. Risks include over-trusting LLM judges and stale datasets."},"roleDetails":{"sales":{"customerProblem":"Agencies run agents without measured evidence of task success, policy compliance or PII leakage.","stakeholders":["AI governance lead","Privacy officer","Agent product owner"],"discoveryQuestions":["What evidence does your AI governance process require for an agent?","Do you have a labeled test set of real requests?","Which failure would be most harmful?"],"valueHypothesis":"Managed evaluation can make agent quality visible and gate releases; evaluator accuracy must be validated.","potentialEngagement":"Set up an evaluation baseline for one production agent.","unsupportedClaims":["Evaluator accuracy","GovCloud availability","Any improvement in agent outcomes"]},"engineering":{"fit":"AgentCore-based agents in Regions where Evaluations is offered.","architectureIntegration":"Evaluators on AgentCore traces; on-demand, online and batch modes.","prerequisites":["AgentCore runtime tracing","Golden dataset","Agreed thresholds"],"constraints":["LLM-judge error","Ground-truth evaluators batch-only","Region availability not stated"],"security":"Include PII leakage and security evaluators; restrict access to evaluation traces containing personal data.","proofOfValue":["Evaluator vs human agreement on a sample","Regression detection on seeded failures"]},"delivery":{"work":"Establish a per-release evaluation gate for one agent.","dependencies":["Dataset","Reviewers","Thresholds"],"ownership":"Agent product owner; governance board approves thresholds.","skillsAdoption":"Teams learn rubric design and judge validation.","governanceCheckpoints":["Threshold approval","Per-criterion release gate","Quarterly dataset refresh"],"acceptanceCriteria":["Proposed: agreed evaluator-human agreement level","Proposed: zero releases with PII-leakage regressions"],"risks":["Over-trust in judges","Stale golden datasets","Trace data exposure"]}},"retrievedAt":"2026-11-11T04:00:04Z","enrichedAt":"2026-11-11T04:05:52Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"No accessibility evaluator is described; agencies would need custom criteria for plain language or accessibility.","procurementImplications":"Pricing not stated in the post.","operatingModelImplications":"Requires owners for golden datasets and evaluation thresholds; evaluation becomes a standing operational function.","sourceVerification":{"openedUrl":"https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/","referenceExcerpt":"A support agent can resolve every ticket and still expose personal information in its replies.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}