{"resourceId":"vals-ai-claude-fable-5-2-independent-benchmarks-weak-tax-legal-20261109","versions":[{"version":"external-cae271e231a4798197f30637bee0329e6e0fe56438c3fb268b7490b9aaaa8660","resource":{"id":"vals-ai-claude-fable-5-2-independent-benchmarks-weak-tax-legal-20261109","title":"Vals AI independent evaluation: Claude Fable 5.2 ranks #1 on Vals Index but trails on Tax Agent Bench (#27 of 84) and Legal Research Bench (#19 of 93)","organization":"Vals AI","sector":"Independent AI evaluation firm","geography":"United States","publishedAt":"November 9, 2026 (model page update)","publicationDate":"2026-11-09","eventDate":"2026-11-09","sourceName":"Vals AI","sourceLabel":"Third-party benchmark results","sourceUrl":"https://www.vals.ai/models/anthropic_claude-fable-5-2","evidenceClass":"independent-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents"],"finding":"Vals AI's independent evaluation ranks Claude Fable 5.2 first of 36 models on its Vals Index (71.31%, High effort, $17.94 per test) and first on finance-agent and coding benchmarks, but shows weaker results on Tax Agent Bench (32.09%, 27th of 84), Legal Research Bench (46.15%, 19th of 93) and the Vals Legal Index (20th of 37). It scored 73.98% on an Immigration benchmark (2nd of 26). Reported latency on the index was 43 minutes 42 seconds.","sledRelevance":"Interpretation: this fills the gap noted in the last edition (no independent testing of Fable 5.2). For state revenue departments, courts, legal aid and immigration-related services, the newest model is not the top performer on tax and legal research benchmarks, so agencies should not assume an upgrade improves those workflows. Long latency also matters for interactive public services.","evidence":"Third-party benchmark scores with confidence intervals across about 20 benchmarks and indices; no public-sector deployment data.","architectureImplications":"Interpretation: model routing by task (for example a different model for tax or legal research) may outperform a single default model.","governanceImplications":"Interpretation: agencies should require domain-specific evaluation before adopting a new default model for legal or tax work.","securityPrivacyImplications":"Not addressed; benchmarks do not test data handling.","caveats":"Benchmarks are Vals-designed, partly proprietary; per-benchmark methodology not inspected; single effort setting; results are not public-sector task evaluations.","streamIds":["anthropic"],"roles":{"sales":"Use independent evidence to set expectations. Vals AI ranks Fable 5.2 first overall but 27th of 84 on tax-agent tasks and 19th of 93 on legal research. For revenue, court and legal-services leaders, ask which tasks they would send to Claude and whether those match the model's strengths. Offer a task-specific evaluation. Do not claim the newest model is best for every workflow or that benchmark ranks predict agency outcomes.","engineering":"Engineering should treat benchmark ranks as routing hints, not guarantees. Fable 5.2 leads coding and finance-agent tasks but trails on tax and legal research, with reported index latency of about 44 minutes at High effort. Build a small agency test set for target workflows, compare Fable 5.2 with incumbent models, and record accuracy, latency and cost. Configure routing so legal or tax tasks can use a better-performing model.","delivery":"Delivery work is an evaluation gate: before changing default models, run agency-specific test sets and compare results with Vals' public scores. Dependencies are curated test cases from subject-matter experts and API access to candidate models. Owner is the agency AI evaluation lead. Risks are adopting a new default that underperforms on legal or tax tasks and slower responses in interactive workflows."},"roleDetails":{"sales":{"customerProblem":"Agencies tend to adopt the newest model as default without checking domain fit for legal, tax or immigration work.","stakeholders":["Agency AI lead","Revenue or tax department director","Court technology officer","Legal services director"],"discoveryQuestions":["Which legal, tax or immigration tasks would use Claude?","Do you have test cases with known answers?","What latency is acceptable?"],"valueHypothesis":"Task-level evaluation could avoid regressions and focus spend on tasks where Fable 5.2 performs well.","potentialEngagement":"A two-week model comparison on agency-curated test sets.","unsupportedClaims":["That Fable 5.2 is best for tax or legal research","That benchmark ranks predict agency accuracy"]},"engineering":{"fit":"Useful for model selection in legal, tax, finance and coding work.","architectureIntegration":"Model routing by task within the agency's Claude or cloud platform.","prerequisites":["Agency test sets","Access to multiple models"],"constraints":["Benchmarks are not public-sector tasks","Single effort setting","Long latency"],"security":"No data-handling testing in source.","proofOfValue":["Compare accuracy on 50 agency tax or legal questions","Measure latency at chosen effort","Record cost per task"]},"delivery":{"work":"Run an evaluation gate before model changes.","dependencies":["Subject-matter expert time","Model access"],"ownership":"Agency AI evaluation lead.","skillsAdoption":"Staff learn to read benchmark reports critically.","governanceCheckpoints":["Evaluation results reviewed before default change","Domain expert sign-off"],"acceptanceCriteria":["Proposed: candidate model meets or exceeds incumbent accuracy on agency test set","Proposed: latency within service target"],"risks":["Domain regressions","Latency","Cost overruns"]}},"retrievedAt":"2026-11-11T04:21:09Z","enrichedAt":"2026-11-11T04:32:28Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: long latency may affect staff workflows that expect quick answers.","procurementImplications":"Interpretation: price per test ($17.94 on the Vals Index) gives a rough cost signal for agentic workloads; contracts should allow model choice by task.","operatingModelImplications":"Interpretation: an evaluation function that tracks third-party and internal results per task should inform model selection.","sourceVerification":{"openedUrl":"https://www.vals.ai/models/anthropic_claude-fable-5-2","referenceExcerpt":"Weaker on HLAB-Vals-1.1 (5.00%, tied #29 of 37) and Tax Agent Bench (32.09%, #27 of 84).","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}