From the NVIDIA edition of November 9, 2026
RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild
Tan Sang Nguyen et al. (26 authors incl. Siddhartha Srinivasa, Dieter Fox, David Hsu) · Robotics evaluation research · Distributed, 124 sites
- Publisher
- arXiv
- Original publication
- Submitted November 2, 2026 (arXiv preprint)
- Source retrieved
- 2026-11-10
What happened
A multi-site preprint evaluates five vision-language-action policies and three frontier AI agents on four embodiments across 5,699 blinded pairwise Open Track comparisons at 124 sites. All systems struggle to complete open-world tasks reliably; Open Track rankings poorly predict post-fine-tuning performance, with rank reversals on all three shared embodiments; rankings agree weakly across embodiment suites (Kendall W = 0.24). After fine-tuning on 100 demonstrations per task, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. On the G1 humanoid no policy completes any of five tasks (average progress 13.3% to 23.2%).
Why it matters
Limited. Relevant as independent context for any university robotics lab or workforce program evaluating NVIDIA's GR00T family; no SLED deployment is evaluated.
Evidence and measured results
Abstract page read on arXiv (v1, 2 Nov 2026) and the PDF text (via pdftotext) checked for its limitations statement. Covers GR00T N1.7, the prior generation, not GR00T 2, and the study states it is an initial snapshot with uneven model and embodiment coverage and five fine-tune tasks per embodiment.
Limitations and uncertainty
Preprint (not peer reviewed). Evaluates GR00T N1.7, not GR00T 2; results may not carry over. Coverage differs across models and embodiments, and the study does not claim exhaustive comparison.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-09; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Labs and programs cannot tell whether vendor-reported robot-policy results will hold on their own robots and tasks.
- Stakeholders
- University robotics faculty
- Research computing director
- Procurement or grants officer
- Workforce program lead
- Discovery questions
- How do you decide which robot policy to adopt?
- Do you track trial counts and partial progress or only success?
- Which embodiments do you actually own?
- Value hypothesis
- A small local evaluation protocol could reduce the risk of adopting a policy on leaderboard evidence alone.
- Potential engagement
- Evaluation-protocol design workshop plus a small blinded comparison on one local robot.
- Unsupported claims
- That GR00T 2 performs better or worse than N1.7
- Any SLED deployment outcome
- That all policies will fail on a given task
- Cost savings
Pre-sales engineering
Role takeaway
- Fit
- Method applies to any lab with a robot and several candidate policies.
- Architecture and integration
- Unified policy serving with standardized protocols across locally operated robots; blinded pairwise trials.
- Prerequisites
- At least one robot embodiment
- Candidate policies with compatible checkpoints
- Demonstration data per task
- Constraints
- Preprint, uneven model and embodiment coverage
- Only five fine-tune tasks per embodiment
- GR00T N1.7, not GR00T 2
- Security
- Not assessed in the paper; local safety interlocks needed for physical trials.
- Proof of value
- Replicate pairwise protocol on one embodiment
- Compare zero-shot versus fine-tuned rankings for rank reversal
- Record partial-progress scores for failed tasks
Delivery
Role takeaway
- Work
- Stand up a local, blinded evaluation harness and run one comparison.
- Dependencies
- Robot access
- Demonstration collection time
- Candidate policy checkpoints
- Ownership
- Lab principal investigator.
- Skills and adoption
- Needs experiment design and robotics data-collection skills.
- Governance checkpoints
- Pre-registered success criteria
- Safety review
- Independent review of results
- Acceptance criteria
- Proposed: comparison table with trial counts and partial-progress scores
- Proposed: documented rank agreement between zero-shot and fine-tuned tests
- Risks
- Small local samples
- Hardware access limits
- Results do not transfer to GR00T 2
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Findings are about model evaluation, not infrastructure; they imply that robotics evaluation needs physical trials on local hardware rather than simulation or leaderboard proxies alone.
Governance
Who approves, reviews and stays accountable for outcomes?
Argues for standardized, blinded, reproducible evaluation before adoption decisions; weak rank agreement means a single benchmark should not drive procurement or grant claims.
Security and privacy
What data, permissions and controls need testing?
Not applicable beyond noting that trials ran on locally operated robots across many sites; no security analysis.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Evaluation at scale relies on many locally operated robots and trained operators; smaller institutions may need partners to run comparable trials.
Procurement
What should contracts, pricing and exit terms secure?
Supports requiring task-specific, reproducible evidence in RFPs and grant proposals rather than leaderboard rankings.
Operating model
Which teams own the service once it runs?
Suggests a shared multi-site evaluation model among institutions; no operating model is prescribed.
Publication history
- 2026-11-09NVIDIA · Issue 655 resources
Stable resource ID: arxiv-robocolosseum-distributed-robot-policy-evaluation-20261102