Observation-contract auditing for valid modality comparison in robot manipulation benchmarks

He, Yuanzhi ORCID: https://orcid.org/0009-0007-8424-2654, Wang, Qiming and Xu, Jiayu 2026. Observation-contract auditing for valid modality comparison in robot manipulation benchmarks. Scientific Reports 10.1038/s41598-026-76821-1

PDF - Accepted Post-Print Version Available under License Creative Commons Attribution. Download (1MB)

Abstract

Robot manipulation benchmarks compare observation modalities such as low-dimensional state, red-green-blue (RGB) images, and point clouds to guide policy design, yet are valid only when each pathway delivers semantically identical state. We show that switching observation pathways silently alters the state content, dropping fields and re-sorting dictionary keys without error or warning, a failure mode we call observation-contract mismatch. We formalize the observation contract as the ordered list of state fields a pathway delivers, define a contract-diff operator, and give a four-step audit that detects mismatches from source code before training. Static analysis of ManiSkill3, RLBench, and the Diffusion Policy robomimic integration shows each silently loses 31 to 53 percent of its observable state. Under one matched protocol the state-only control (0.606 mean success) matches a default-pathway Diffusion Policy (0.600) and beats a retrained action chunking transformer (ACT) baseline (0.400), since the state pathway’s contract is already complete. A contract-controlled factorial crossing state completeness with point-cloud availability then shows that removing object geometry collapses mean success from 0.606 to 0.016, while point clouds neither rescue it nor help under full state. Within the tested regime the state contract dominates policy success, making observation-contract auditing a prerequisite for credible modality comparison

Item Type:	Article
Date Type:	Published Online
Status:	In Press
Schools:	Schools > Computational & Mathematical Sciences Schools > Computer Science & Informatics
Publisher:	Nature Research
ISSN:	2045-2322
Date of First Compliant Deposit:	29 October 2026
Date of Acceptance:	15 October 2026
Last Modified:	29 Oct 2026 14:30
URI:	https://orca.cardiff.ac.uk/id/eprint/190526

Actions (repository staff only)

Edit Item