When recalling information in conversation, people often arrive at the recollection after multiple turns. However, existing benchmarks for evaluating agent capabilities in such tip-of-the-tongue search processes are restricted to single-turn settings. To more realistically simulate tip-of-the-tongue search, we introduce Dual-agent based Evaluation Through Obscure Under-specified Retrieval (DETOUR), a dual-agent evaluation benchmark containing 1,011 prompts. The benchmark design involves a Primary Agent, which is the subject of evaluation, tasked with identifying the recollected entity through querying a Memory Agent that is held consistent across evaluations. Our results indicate that current state-of-the-art models still struggle with our benchmark, only achieving 36% accuracy when evaluated on all modalities (text, image, audio, and video), highlighting the importance of enhancing capabilities in underspecified scenarios.
Are Language Models Efficient Reasoners? A Perspective from Logic Programming
ETH Zürich（苏黎世联邦理工学院）；MPI for Intelligent Systems, Tübingen（智能系统马克斯·普朗克研究所）；EPFL（苏黎世联邦理工学院）；Idiap Research Institute（Idiap研究机构）；Purdue University（普渡大学）
Modeling Grammatical Hypothesis Testing in Young Learners: A Sequence-Based Learning Analytics Study of Morphosyntactic Reasoning in an Interactive Game
Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）；MiLM Plus, Xiaomi Inc.（小米公司MiLM Plus团队）；School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Data Science, Fudan University（复旦大学数据科学学院）
DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards
York University（约克大学）；RBC；Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；Nanyang Technological University（南洋理工大学）；Salesforce Research（Salesforce研究）
Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Alibaba Group（阿里巴巴集团）；DAMO Academy, Alibaba Group（阿里巴巴集团大模型学院）；Nanjing Institute of Software Technology（南京软件技术研究所）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Hohai University（河海大学）
Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
Tsinghua University（清华大学）；Chongqing University（重庆大学）；Peking University（北京大学）；ZenoMind AI；Xi’an Jiaotong University（西安交通大学）；Beijing Institute of Technology（北京理工大学）；Southeast University（东南大学）；Shanghai Jiao Tong University（上海交通大学）；Joy Future Academy（京东探索研究院）；The University of Hong Kong（香港大学）
Reasoning While Asking: Transforming Reasoning Large Language Models from Passive Solvers to Proactive Inquirers
National Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；Artificial Intelligence Research Institute（人工智能研究院）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of Texas, Dallas（德克萨斯大学达拉斯分校）；University of Minnesota（明尼苏达大学）
OralAgent: Integrating Reasoning, Tools, and Knowledge for Interactive Dental Image Analysis
Faculty of Dentistry, the University of Hongkong, Hong Kong SAR, China（香港大学牙科学院，中国香港特别行政区）；Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, USA（匹兹堡大学电气与计算机工程系，美国宾夕法尼亚州匹兹堡）；Shenzhen University, China（深圳大学，中国）；Department of Craniomaxillofacial Surgery, Shanghai Ninth People’s Hospital, China（上海第九人民医院口腔颌面外科部，中国）；Nanyang technological University, Singapore（南洋理工大学，新加坡）；School of Biomedical Engineering, Southern Medical University, China（南方医科大学生物医学工程学院，中国）；Singapore University of Technology and Design, Singapore（新加坡科技设计大学，新加坡）；University of Auckland, new zealand（奥克兰大学，新西兰）；Shanghai Artificial Intelligence Laboratory , China（上海人工智能实验室，中国）
