This paper introduces AQA-Bench, a novel benchmark to assess the sequential reasoning capabilities of large language models (LLMs) in algorithmic contexts, such as depth-first search (DFS). The key feature of our evaluation benchmark lies in its interactive evaluation protocol - for example, in DFS, the availability of each node's connected edge is contingent upon the model's traversal to that node, thereby necessitating the LLM's ability to effectively remember visited nodes and strategize subsequent moves considering the possible environmental feedback in the future steps. We comprehensively build AQA-Bench with three different algorithms, namely binary search, depth-first search, and breadth-first search, and to evaluate the sequential reasoning ability of 14 different LLMs. Our investigations reveal several interesting findings: (1) Closed-source models like GPT-4 and Gemini generally show much stronger sequential reasoning ability, significantly outperforming open-source LLMs. (2) Naively providing in-context examples may inadvertently hurt few-shot performance in an interactive environment due to over-fitting to examples. (3) Instead of using optimal steps from another test case as the in-context example, a very limited number of predecessor steps in the current test case following the optimal policy can substantially boost small models' performance. (4) The performance gap between weak models and strong models is greatly due to the incapability of weak models to start well. (5) The scaling correlation between performance and model size is not always significant, sometimes even showcasing an inverse trend. We hope our study can catalyze future work on advancing the understanding and enhancement of LLMs' capabilities in sequential reasoning. The code is available at https://github.com/UCSC-VLAA/AQA-Bench.
Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education（教育部新一代人工智能技术及其交叉应用关键实验室（东南大学））；The University of Manchester（曼彻斯特大学）；The University of Edinburgh（爱丁堡大学）；Law and Innovation Lab, School of Law, Southeast University（东南大学法学院创新与法律实验室）；State Key Lab. for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Monash University（莫纳什大学）
FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment
School of Biomedical Engineering (Suzhou), USTC（中国科学技术大学苏州生物医学工程学院）；Suzhou Institute of Biomedical Engineering and Technology, CAS（中国科学院苏州生物医学工程技术研究所）；Institute of High-Performance Computing, A*STAR, Singapore（新加坡科技研究局高性能计算研究所）；School of Psychology, Beijing Sports University（北京体育大学心理学院）；School of Competitive Sports, Beijing Sports University（北京体育大学竞技体育学院）；Department of Robotics, University of Michigan（密歇根大学机器人系）
Stanford University（斯坦福大学）；Robotics at Google（谷歌机器人部门）
We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interaction with objects is allowed and even encouraged to accomplish a task. For example, the robot can move objects if needed in order to clear a path leading to the goal location. Our benchmark comprises two novel elements: 1) a new experimental setup, the Interactive Gibson Environment (iGibson 0.5), which simulates high fidelity visuals of indoor scenes, and high fidelity physical dynamics of the robot and common objects …
Can Vision-Language Models Judge Olympic Diving? From Reasoning to Scores in Zero-Shot Action Quality Assessment
ESPOL Polytechnic University（ESPOL理工大学）；University of Granada（格拉纳达大学）；Universidad de Las Palmas de Gran Canaria（拉斯帕尔马斯·德·大加那利大学）；Michigan Technological University（密歇根理工大学）
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Shenzhen Institute of Advanced Technology Chinese Academy of Sciences（深圳先进技术研究院中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Dalian University of Technology（大连理工大学）；UNSW Sydney（悉尼大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors
Center of Excellence for Generative AI, KAUST（生成人工智能卓越中心，KAUST）；The Swiss AI Lab, IDSIA-USI/SUPSI（瑞士人工智能实验室，IDSIA-USI/SUPSI）；NNAISENSE Principia-AI/PhysGym（NNAISENSE 原初人工智能/PhysGym）
Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives
University of Macau（澳门大学）；Westlake University（西湖大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Cambridge（剑桥大学）；University of Aberdeen（阿伯丁大学）
SEAR: Spoofing Evidence-Grounded Audio Reasoning Benchmark for Audio Language Models
University of Surrey（萨里大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Guangxi University（广西大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
Tsinghua University（清华大学）；Chongqing University（重庆大学）；Peking University（北京大学）；ZenoMind AI；Xi’an Jiaotong University（西安交通大学）；Beijing Institute of Technology（北京理工大学）；Southeast University（东南大学）；Shanghai Jiao Tong University（上海交通大学）；Joy Future Academy（京东探索研究院）；The University of Hong Kong（香港大学）
IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）智能科学与工程学院）；Zhejiang University（浙江大学）；State Key Lab of CAD&CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）
Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping
The Chinese University of Hong Kong, China（香港中文大学）；Singapore Management University, Singapore（新加坡管理学院）；Renmin University of China, China（中国人民大学）；Tsinghua University, China（清华大学）；Southwest University, China（西南大学）
KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Xiamen University Malaysia（马来西亚厦门大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
NVIDIA；University of Maryland, College Park（马里兰大学 College Park 分校）；University of Science and Technology of China（中国科学技术大学）；Seoul National University（首尔国立大学）；Adobe
Inter3D: A Benchmark and Strong Baseline for Human-Interactive 3D Object Reconstruction
Shenzhen University（深圳大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Shanghai AI Laboratory（上海人工智能实验室）
针对人类交互式多部件物体离散状态重建需指数级模型的问题，本文提出 Inter3D 基准、自采数据集与评估流程，并以 Space Discrepancy Tensors、互状态正则化和占据网格采样构建强基线。
Words to World: Benchmarking LLMs in Interactive Physical Environments
Michigan State University（密歇根州立大学）；Zhejiang University（浙江大学）；Microsoft（微软）；University of Rochester（罗切斯特大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests
University of Trento（特伦托大学）；ILCC University of Edinburgh（爱丁堡大学ILCC学院）；UCL（伦敦大学学院）；Saarland University（萨尔兰大学）；Heriot-Watt University（赫瑞-瓦特大学）；University of Potsdam（波茨坦大学）；DFKI（德国达姆施塔特研究所）；University of Amsterdam（阿姆斯特丹大学）；Faculty of Engineering, Free University of Bozen-Bolzano（博兹纳-博尔扎诺自由大学工程学院）
University of Texas Austin（德克萨斯大学奥斯汀分校）；California Institute of Technology（加州理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research（微软研究院）；Northwestern University（西北大学）；University of Cambridge（剑桥大学）
Evaluating the performance of human is a common need across many applications, such as in engineering and sports. When evaluating human performance in completing complex and interactive tasks, the most common way is to use a metric having been proved efficient for that context, or to use subjective measurement techniques. However, this can be an error prone and unreliable process since static metrics cannot capture all the complex contexts associated with such tasks and biases exist in subjective measurement. The objective of our research is to create data-driven AI agents as computational ben…
