TransportationGames：对（多模态）大型语言模型交通知识的基准测试
TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models
AI总结：
本文提出交通领域评估基准TransportationGames，基于Bloom分类法测试(M)LLMs的记忆、理解和应用能力，实验表明模型整体表现仍有较大改进空间，为该领域后续研究奠定基础。
摘要：
大型语言模型（LLMs）和多模态大型语言模型（MLLMs）展现出了卓越的通用能力，甚至在法律、经济、交通和医学等专业领域也表现出适应性。目前，已提出许多特定领域的基准来验证(M)LLMs在特定领域的性能。在众多领域中，交通在现代社会中扮演着至关重要的角色，因为它影响着经济、环境以及数十亿人的生活质量。然而，目前尚不清楚(M)LLMs掌握了多少交通知识，以及它们能否可靠地执行与交通相关的任务。为了填补这一空白，我们提出了TransportationGames，这是一个经过精心设计且全面的评估基准，用于评估(M)LLMs在交通领域的表现。通过综合考虑现实场景中的应用，并参考Bloom分类法的前三个层次，我们通过所选任务测试了各种(M)LLMs在记忆、理解和应用交通知识方面的表现。实验结果表明，尽管一些模型在某些任务中表现良好，但总体而言仍有很大的改进空间。我们希望TransportationGames的发布能为未来的研究奠定基础，从而加速(M)LLMs在交通领域的实施与应用。
英文摘要：
Large language models (LLMs) and multimodal large language models (MLLMs) have shown excellent general capabilities, even exhibiting adaptability in many professional domains such as law, economics, transportation, and medicine. Currently, many domain-specific benchmarks have been proposed to verify the performance of (M)LLMs in specific fields. Among various domains, transportation plays a crucial role in modern society as it impacts the economy, the environment, and the quality of life for billions of people. However, it is unclear how much traffic knowledge (M)LLMs possess and whether they can reliably perform transportation-related tasks. To address this gap, we propose TransportationGames, a carefully designed and thorough evaluation benchmark for assessing (M)LLMs in the transportation domain. By comprehensively considering the applications in real-world scenarios and referring to the first three levels in Bloom's Taxonomy, we test the performance of various (M)LLMs in memorizing, understanding, and applying transportation knowledge by the selected tasks. The experimental results show that although some models perform well in some tasks, there is still much room for improvement overall. We hope the release of TransportationGames can serve as a foundation for future research, thereby accelerating the implementation and application of (M)LLMs in the transportation domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
