Benchmark-path fidelity in parallel robot-policy evaluation: a CALVIN case study

He, Yuanzhi ORCID: https://orcid.org/0009-0007-8424-2654, Wang, Qiming and Xu, Jiayu 2026. Benchmark-path fidelity in parallel robot-policy evaluation: a CALVIN case study. Journal of Supercomputing 82 (16) , 810. 10.1007/s11227-026-08965-w

PDF - Published Version Available under License Creative Commons Attribution. Download (561kB)

Abstract

Evaluating long-horizon robot manipulation policies is a high-throughput computing workload: one CALVIN-style campaign sweeps checkpoints, seeds and task sequences across many concurrent simulator workers, consuming tens of thousands of rollouts before any score is inspected. Evaluation-path correctness is therefore a workflow-validation concern in parallel evaluation: an adapter defect shared by workers can yield a well-formed table of zeros while scheduler and determinism checks pass. We report a CALVIN case study and use it to propose the Benchmark-Fidelity Diagnostic Protocol (BFDP): a cheap-first set of single-worker controls, run before a parallel sweep, that audits export horizon, teacher replay semantics and reset fidelity in a fixed order. On Composing Actions from Language and Vision (CALVIN) scene D with a vision-language-to-diffusion pipeline, BFDP separates three findings of three different kinds. For the two tasks with a measured effect onset, the fixed 8-step export ends before the observed onset, and across benchmark-critical tasks it retains only 12–16% of the mean teacher-episode length. This is a confirmed defect of the export path: across all 14 episodes, in a paired intervention with export horizon as the only changed factor, exact-reset teacher replay rises from 0/14 to 14/14. The symbolic path returns one canonical robot state for all 64 probed initial conditions; this is the benchmark’s documented neutral reset rather than a defect, and the resulting exact-versus-symbolic split measures the sensitivity of our open-loop teacher control to its start state. Exact continuous replay reaches 1/3 on an official three-subtask probe, which bounds what teacher replay can establish under the official chained protocol rather than indicating a further defect. The full set of controls is between 0.36 and 1.8% of the subtask-rollout workload of the campaign it precedes, depending on how far the policy gets before each official sequence terminates. We present BFDP as a proposed diagnostic checklist demonstrated on one benchmark, one scene and one policy family, not as a validated general standard.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Computational & Mathematical Sciences Schools > Computer Science & Informatics
Publisher:	Springer
ISSN:	0920-8542
Date of First Compliant Deposit:	27 October 2026
Date of Acceptance:	14 October 2026
Last Modified:	27 Oct 2026 12:30
URI:	https://orca.cardiff.ac.uk/id/eprint/190452

Actions (repository staff only)

Edit Item