In this paper, we explore the capabilities of state-of-the-art large language models (LLMs) such as GPT-4, GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3, and Llama 3.1 in solving some selected undergraduate-level transportation engineering problems. We introduce TransportBench, a benchmark dataset that includes a sample of transportation engineering problems on a wide range of subjects in the context of planning, design, management, and control of transportation systems. This dataset is used by human experts to evaluate the capabilities of various commercial and open-sourced LLMs, especially their accuracy, consistency, and reasoning behaviors, in solving transportation engineering problems. Our comprehensive analysis uncovers the unique strengths and limitations of each LLM, e.g. our analysis shows the impressive accuracy and some unexpected inconsistent behaviors of Claude 3.5 Sonnet in solving TransportBench problems. Our study marks a thrilling first step toward harnessing artificial general intelligence for complex transportation challenges.
A Survey and Benchmarking of Spatial-Temporal Traffic Data Imputation Models
Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education, and School of Computer Science and Technology, Beijing Jiaotong University（大数据与人工智能交通运输关键实验室、教育部，计算机科学与技术学院，北京交通大学）；Department of Computer Science, Aalborg University（计算机科学系，奥尔堡大学）；Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University（北京交通大数据挖掘与具身智能关键实验室，计算机科学与技术学院，北京交通大学）
TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation
School of Transportation（交通学院）；Southeast University（东南大学）；School of Artificial Intelligence and Computer Science（人工智能与计算机科学学院）；Jiangnan University（江南大学）；Department of Civil and Environmental Engineering（土木与环境工程系）；Hong Kong Polytechnic University（香港理工大学）
CogRail: Benchmarking VLMs in Cognitive Intrusion Perception for Intelligent Railway Transportation Systems
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）；Signal & Communication Research Institute, China Academy of Railway Sciences（信号与通信研究所，中国铁路科学研究院）；Beijing Huairou Academy of Parallel Sensing（北京怀柔平行感知院）；School of Computer Science and Technology, Beijing Jiaotong University（计算机科学与技术学院，北京交通大学）；State Key Lab of Intelligent Transportation Systems, School of Transportation Science and Engineering, Beihang University（智能交通系统国家重点实验室，交通科学与工程学院，北京航空航天大学）
Towards Computational Performance Engineering for Unsupervised Concept Drift Detection -- Complexities, Benchmarking, Performance Analysis
Center for Interdisciplinary Digital Sciences (CIDS), Technische Universität Dresden（德累斯顿工业大学跨学科数字科学中心（CIDS））；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig（德累斯顿/莱可堡可扩展数据分析与人工智能中心（ScaDS.AI））；Chair of Computer Graphics and Visualization (CGV), Technische Universität Dresden（德累斯顿工业大学计算机图形与可视化讲座）；Center for Information Services and High Performance Computing (ZIH), Technische Universität Dresden（德累斯顿工业大学信息服务与高性能计算中心（ZIH））
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tsinghua University（清华大学）；Peking University（北京大学）；Southeast University（东南大学）
Forecasting MBTA Transit Dynamics: A Performance Benchmarking of Statistical and Machine Learning Models
Sunset High School（Sunset 高中）；Woodbridge High School（Woodbridge 高中）；Western Academy of Beijing（北京西方学院）；Computer Science Department, Metropolitan College（Metropolitan College 计算机科学系）
Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support
Bourns College of Engineering, Center for Environmental Research and Technology, University of California at Riverside, CA, USA（伯恩斯工程学院，环境研究与技术中心，加州大学河滨分校，美国，加利福尼亚州河滨）
AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving
Texas A&M University（德州农工大学）；University of Toronto（多伦多大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Maryland（马里兰大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）
Benchmarking Machine Learning Models for IoT Malware Detection under Data Scarcity and Drift
Department of Computer Science, The College of Wooster, Wooster, USA（沃斯特学院计算机科学系）；Department of Computer Science and Engineering, University of Nevada-Reno, Reno, USA（内华达大学雷诺分校计算机科学与工程系）
New Benchmarking Shows Limited Generalization Power of TCR Antigenic Epitope Prediction Models
Trustworthy and Intelligent Computing Lab (TAIC), Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County（可信智能计算实验室（TAIC），计算机科学与电气工程系，马里兰大学巴尔的摩分校）；Children’s Hospital of Philadelphia（费城儿童医院）；Department of Bioengineering, University of Pennsylvania（生物工程系，宾夕法尼亚大学）；Institute for Immunology & Immune Health, University of Pennsylvania（免疫学与免疫健康研究所，宾夕法尼亚大学）；Institute for RNA Innovation, University of Pennsylvania（RNA创新研究所，宾夕法尼亚大学）；Abramson Cancer Center, University of Pennsylvania（Abramson癌症中心，宾夕法尼亚大学）；Center for Precision Engineering for Health, University of Pennsylvania（健康精准工程中心，宾夕法尼亚大学）；Center for Cellular Immunotherapies, University of Pennsylvania（细胞免疫治疗中心，宾夕法尼亚大学）
Scenario Engineering for Autonomous Transportation: A New Stage in Open-Pit Mines
Hong Kong Baptist University（香港浸会大学）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）；Peng Cheng Laboratory（鹏城实验室）；Indiana University-Purdue University Indianapolis (IUPUI)（印第安纳大学-普渡大学印第安纳波利斯联合分校）；Guangdong Provincial Key Laboratory IRADS（广东省IRADS重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Scenarios Engineering driven Autonomous Transportation in Open-Pit Mines
Hong Kong Baptist University（香港浸会大学）；IRADS, BNU-HKBU UIC（北京师范大学-香港浸会大学联合国际学院IRADS）；Peng Cheng Laboratory（鹏城实验室）；Indiana University-Purdue University Indianapolis（印第安纳大学与普渡大学印第安纳波利斯联合分校）；University of Chinese Academy of Sciences（中国科学院大学）；Waytous Ltd.（万道科技有限公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
