Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Risks

Meng, Xianhui, Zhang, Yuchen, Huang, Zhijian, Lu, Zheng, Ji, Ziling, Lin, Yandan, Yin, Yaoyao, Zhang, Hongyuan, Zhou, Wei, Jiang, Guangfeng, Zhang, Li, Chen, Long, Ye, Hangjun, Liu, Jun and Hao, Xiaoshuai 2026. DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Risks. Presented at: MM '26: The 34th ACM International Conference on Multimedia, Rio de Janeiro, Brazil, 10 - 14 November 2026. MM '26: Proceedings of the 34th ACM International Conference on Multimedia. ACM International Conference on Multimedia (ACM MM). New York: ACM, pp. 16918-16925. 10.1145/3767308.3838641

[thumbnail of 3767308.3838641.pdf] PDF - Published Version
Available under License Creative Commons Attribution.

Download (3MB)

Abstract

Driving safety in assisted driving systems requires holistic understanding of both the external traffic environment and the internal driver state. However, existing research typically treats out-cabin scene understanding and in-cabin driver monitoring as separate problems, resulting in fragmented safety assessment and limited support for real-world warning applications. Moreover, conventional task-specific models often perform well on narrowly defined tasks but struggle to generalize to the diverse and long-tail hazards encountered in open-world driving scenarios. To address these limitations, we propose a new research setting, termed driving safety understanding for assisted driving, which aims to jointly perceive, reason about, and warn against risks by integrating information from both inside and outside the vehicle. To support this task, we construct a large-scale driving safety dataset containing 98K vision-language annotations that cover diverse in-cabin and out-cabin safety-critical scenarios, and further curate a 3K benchmark named DSBench(Driving Safety Benchmark) set for systematic evaluation. Based on this dataset, we develop DSVLM, a domain-specialized vision-language model for driving safety. Unlike end-to-end assisted driving models, DSVLM is designed as an in-vehicle safety assistant that identifies risk factors, infers potential danger, and provides timely warnings to help prevent accidents. Extensive evaluations against a wide range of mainstream open-source and proprietary VLMs show that existing models suffer substantial performance degradation in complex safety-critical situations, revealing a clear gap in reliable driving-safety understanding. In contrast, DSVLM establishes a new state of the art, achieving an average score of 68.4, which surpasses the strongest commercial baseline, Seed-1.6 (49.52), by 18.88 points. Its superiority is particularly pronounced on highly challenging tasks such as cockpit understanding, where DSVLM attains 80.1, dramatically outperforming the runner-up score of 29.49. These results demonstrate that DSVLM not only improves overall performance, but also delivers markedly stronger robustness and safety-awareness in complex real-world driving scenarios. The benchmark toolkit, source code, and model checkpoints are publicly available at https://github.com/mengxh20/DSBench.

Item Type: Conference or Workshop Item - published (Paper)
Date Type: Published Online
Status: Published
Schools: Schools > Computational & Mathematical Sciences
Schools > Computer Science & Informatics
Publisher: ACM
ISBN: 9798400722134
Date of First Compliant Deposit: 13 November 2026
Last Modified: 13 Nov 2026 12:00
URI: https://orca.cardiff.ac.uk/id/eprint/190900

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics