Remarkable performance of large language models (LLMs) in a variety of tasks brings forth many opportunities as well as challenges of utilizing them in production settings. Towards practical adoption of LLMs, multi-agent systems hold great promise to augment, integrate, and orchestrate LLMs in the larger context of enterprise platforms that use existing proprietary data and models to tackle complex real-world tasks. Despite the tremendous success of these systems, current approaches rely on narrow, single-focus objectives for optimization and evaluation, often overlooking potential constraints in real-world scenarios, including restricted budgets, resources and time. Furthermore, interpreting, analyzing, and debugging these systems requires different components to be evaluated in relation to one another. This demand is currently not feasible with existing methodologies. In this postion paper, we introduce the concept of reasoning capacity as a unifying criterion to enable integration of constraints during optimization and establish connections among different components within the system, which also enable a more holistic and comprehensive approach to evaluation. We present a formal definition of reasoning capacity and illustrate its utility in identifying limitations within each component of the system. We then argue how these limitations can be addressed with a self-reflective process wherein human-feedback is used to alleviate shortcomings in reasoning and enhance overall consistency of the system.
RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Research Centre for Data Science & Artificial Intelligence(数据科学与人工智能研究中心);Department of Computer and Data Sciences, Case Western Reserve University(凯斯西储大学计算机与数据科学系)
Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Models
LARG, Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology(LARG,社会计算与交互机器人研究中心,哈尔滨工业大学);School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学);The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);ByteDance Seed (China)(字节跳动种子(中国))
Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models
University of California, Berkeley(加州大学伯克利分校);University of Cambridge(剑桥大学);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Tokyo(东京大学)
MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering
National Library of Medicine, Division of Intramural Research(国家医学图书馆,院内研究部);University of Illinois at Urbana-Champaign, Department of Computer Science(伊利诺伊大学厄巴纳-香槟分校计算机科学系);University of Michigan Medical School(密歇根大学医学院)
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);University of California, Berkeley(加州大学伯克利分校);Boston University(波士顿大学);MIT-IBM Watson AI(麻省理工学院-IBM沃森人工智能实验室)
人类有非凡的能力进行溯因推理,并对图像的字面内容之外的内容进行假设。通过识别分散在场景中的具体视觉线索,我们几乎忍不住根据我们对世界的日常经验和知识,在文字场景之外做出可能的推断。例如,如果我们在路边看到“20英里/小时”的标志,我们可能会认为这条街位于住宅区(而不是高速公路上),即使没有房屋被拍到。机器能执行类似的视觉推理吗?我们提出了夏洛克,103 K图像的注释语料库,用于测试机器的溯因推理能力超出字面图像内容。我们采用自由观看范式:参与者首先观察并识别图像中的显著线索(例如,物体,动作),然后根据线索对场景进行合理的推断。总的来说,我们收集了363 K(线索,推理)对,形成了第一个溯因视觉推理数据集。使用我们的语料库,我们测试三个互补轴的溯因推理。我们评估模型的能力:i)从大型候选语料库中检索相关推断; ii)通过边界框定位推断的证据; iii)比较合理的推断,以匹配新收集的19 K Likert尺度判断的诊断语料库上的人类判断。虽然我们发现使用多任务目标微调CLIP-RN 50 x64的性能优于强大的基线,但模型性能和人类一致性之间存在显着的空间。数据、模型和排行榜可在http://visualabduction.com/获得
Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R$^2$)GRPO
HKUST Hong Kong SAR, China(香港科技大学);SEU Jiangsu, China(江苏大学);HKUST(GZ) Guangzhou, China(香港科技大学(广州));Zhipu AI Beijing, China(智谱AI);HKUST(GZ), HKUST Guangzhou, China(香港科技大学(广州)、香港科技大学)