Vals AI independent evaluation: Claude Fable 5.2 ranks #1 on Vals Index but trails on Tax Agent Bench (#27 of 84) and Legal Research Bench (#19 of 93)

Vals AI · Independent AI evaluation firm · United States

Publisher: Vals AI
Original publication: November 9, 2026 (model page update)
Source retrieved: 2026-11-11
Event date: 2026-11-09

What happened

Vals AI's independent evaluation ranks Claude Fable 5.2 first of 36 models on its Vals Index (71.31%, High effort, $17.94 per test) and first on finance-agent and coding benchmarks, but shows weaker results on Tax Agent Bench (32.09%, 27th of 84), Legal Research Bench (46.15%, 19th of 93) and the Vals Legal Index (20th of 37). It scored 73.98% on an Immigration benchmark (2nd of 26). Reported latency on the index was 43 minutes 42 seconds.

Why it matters

This fills the gap noted in the last edition (no independent testing of Fable 5.2). For state revenue departments, courts, legal aid and immigration-related services, the newest model is not the top performer on tax and legal research benchmarks, so agencies should not assume an upgrade improves those workflows. Long latency also matters for interactive public services.

Evidence and measured results

Third-party benchmark scores with confidence intervals across about 20 benchmarks and indices; no public-sector deployment data.

Limitations and uncertainty

Benchmarks are Vals-designed, partly proprietary; per-benchmark methodology not inspected; single effort setting; results are not public-sector task evaluations.

Put this evidence to work

Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-11; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.

Sales

Role takeaway

Customer problem: Agencies tend to adopt the newest model as default without checking domain fit for legal, tax or immigration work.
Stakeholders: Agency AI lead
Revenue or tax department director
Court technology officer
Legal services director
Discovery questions: Which legal, tax or immigration tasks would use Claude?
Do you have test cases with known answers?
What latency is acceptable?
Value hypothesis: Task-level evaluation could avoid regressions and focus spend on tasks where Fable 5.2 performs well.
Potential engagement: A two-week model comparison on agency-curated test sets.
Unsupported claims: That Fable 5.2 is best for tax or legal research
That benchmark ranks predict agency accuracy

Pre-sales engineering

Role takeaway

Fit: Useful for model selection in legal, tax, finance and coding work.
Architecture and integration: Model routing by task within the agency's Claude or cloud platform.
Prerequisites: Agency test sets
Access to multiple models
Constraints: Benchmarks are not public-sector tasks
Single effort setting
Long latency
Security: No data-handling testing in source.
Proof of value: Compare accuracy on 50 agency tax or legal questions
Measure latency at chosen effort
Record cost per task

Delivery

Role takeaway

Work: Run an evaluation gate before model changes.
Dependencies: Subject-matter expert time
Model access
Ownership: Agency AI evaluation lead.
Skills and adoption: Staff learn to read benchmark reports critically.
Governance checkpoints: Evaluation results reviewed before default change
Domain expert sign-off
Acceptance criteria: Proposed: candidate model meets or exceeds incumbent accuracy on agency test set
Proposed: latency within service target
Risks: Domain regressions
Latency
Cost overruns

Implementation considerations

Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.

Architecture and integration

What must connect, and where does the AI sit in the workflow?

Model routing by task (for example a different model for tax or legal research) may outperform a single default model.

Governance

Who approves, reviews and stays accountable for outcomes?

Agencies should require domain-specific evaluation before adopting a new default model for legal or tax work.

Security and privacy

What data, permissions and controls need testing?

Not addressed; benchmarks do not test data handling.

Accessibility and workforce

Who is affected, and what skills or accommodations follow?

Long latency may affect staff workflows that expect quick answers.

Procurement

What should contracts, pricing and exit terms secure?

Price per test ($17.94 on the Vals Index) gives a rough cost signal for agentic workloads; contracts should allow model choice by task.

Operating model

Which teams own the service once it runs?

An evaluation function that tracks third-party and internal results per task should inform model selection.

Publication history

2026-11-10Anthropic · Issue 1210 resources

Read preserved resource versions (JSON)

Stable resource ID: vals-ai-claude-fable-5-2-independent-benchmarks-weak-tax-legal-20261109