TRIP-Evaluate: 一个用于评估交通领域大模型的开放多模态基准
TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation
AI总结：
TRIP-Evaluate是首个开放多模态交通评估基准，通过837项任务覆盖车辆、交通管理、旅行者和规划设计功能，提供能力、模态和难度标签，支持跨模态诊断，揭示大模型在多步工程计算、规则约束推理和多模态场景理解方面的不足。
摘要：
大型语言模型（LLMs）和多模态大模型（MLLMs）日益用于交通任务，如法规问答、交通管理支持、工程审查和自动驾驶场景推理。然而，交通流程规则密集、计算密集、安全关键且本质上多模态。现有通用基准难以证明模型能否正确应用法规、执行可验证的工程计算或可靠解读交通场景，而公开的交通基准范围狭窄且不支持文本、图像和点云数据的细粒度诊断。为此，我们提出TRIP-Evaluate，一个开放的多模态交通大模型评估基准。该基准采用角色-任务-知识分类法组织837项任务，每项任务标注能力、模态和难度标签，支持从整体准确率到特定失败模式的诊断。当前版本包含596个文本任务、198个图像任务和43个点云任务。TRIP-Evaluate还标准化了任务构建、质量控制、提示、解码和评分，以提高跨模型可比性。在多样化的模型面板上结果表明，基于文本的性能在提升，但多步工程计算、规则约束推理、多模态场景理解和点云理解仍存在显著缺陷。总体而言，TRIP-Evaluate为模型选择、回归测试和交通应用的安全部署提供了可重复、可诊断和工程对齐的评估基线。
英文摘要：
Large language models (LLMs) and multimodal large models (MLLMs) are increasingly used for transportation tasks such as regulation question answering, traffic management support, engineering review, and autonomous-driving scene reasoning. Yet transportation workflows are rule-intensive, computation-intensive, safety-critical, and inherently multimodal. Existing general benchmarks provide limited evidence of whether a model can apply regulations correctly, perform verifiable engineering calculations, or interpret traffic scenes reliably, while the small number of public transportation benchmarks remain narrow in scope and rarely support fine-grained diagnosis across text, images, and point-cloud data. To address this gap, we present TRIP-Evaluate, an open multimodal benchmark for large models in transportation. The benchmark organizes 837 items using a role-task-knowledge taxonomy that covers vehicle, traffic-management, traveler, and planning-and-design functions. Each item is annotated with capability, modality, and difficulty labels, enabling diagnosis from overall accuracy down to specific failure modes. The current release includes 596 text items, 198 image items, and 43 point-cloud items. TRIP-Evaluate also standardizes item construction, quality control, prompting, decoding, and scoring to improve cross-model comparability. Results on a diverse panel of models show that text-based performance is improving, but substantial weaknesses remain in multi-step engineering calculation, rule-constrained reasoning, multimodal scene understanding, and point-cloud understanding. Overall, TRIP-Evaluate provides a reproducible, diagnosable, and engineering-aligned evaluation baseline for model selection, regression testing, and safer deployment in transportation applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
