Combinatorial Reasoning: Selecting Reasons in Generative AI Pipelines via Combinatorial Optimization
Icosa Computing Inc.(伊科萨计算公司);NASA ARC(美国国家航空航天局艾姆斯研究中心);USRA Research Institute for Advanced Computer Science (RIACS)(美国大学空间研究协会高级计算机科学研究所);Hewlett Packard Labs(惠普实验室);DataStax(DataStax公司)
Mert Esencan, Tarun Advaith Kumar, Ata Akbari Asanjan, P. Aaron Lott, Masoud Mohseni, Can Unlu, Davide Venturelli, Alan Ho
本文提出完全自动化的组合推理框架 CR,将 LLM 流水线采样的理由建模为 QUBO 问题并选择子集构建思维链提示,实验表明组合优化可增强生成式 AI 推理。
摘要:
近期的大语言模型(LLM)在需要人类智能的任务上展示了令人瞩目的能力,并向着类人人工智能(AI)迈出了重要一步。然而,LLM 在推理任务上的表现一直不够理想,且其推理能力仍存在重大争议。尽管已有研究表明,面向 LLM 的提示技术选择会改变其在包括推理在内的众多任务上的表现,但性能最佳的技术需要在了解当前任务的情况下由人工编写提示。我们提出了一个称为组合推理(Combinatorial Reasoning,CR)的框架,这是一种完全自动化的提示方法:从 LLM 流水线中采样理由,并将其映射为二次无约束二值优化(Quadratic Unconstrained Binary Optimization,QUBO)问题。该框架研究 QUBO 的解能否被有效用于选择一个有用的理由子集,以构建思维链风格的提示。我们探索了利用专用求解器加速 CR 的方法。我们还研究了更简单的零样本策略的性能,例如线性多数规则或随机选择理由。我们的初步研究表明,将组合求解器与生成式 AI 流水线相结合是 AI 推理的一条有吸引力的途径,并阐明了未来 CR 方法的设计原则。
英文摘要:
Recent Large Language Models (LLMs) have demonstrated impressive capabilities at tasks that require human intelligence and are a significant step towards human-like artificial intelligence (AI). Yet the performance of LLMs at reasoning tasks have been subpar and the reasoning capability of LLMs is a matter of significant debate. While it has been shown that the choice of the prompting technique to the LLM can alter its performance on a multitude of tasks, including reasoning, the best performing techniques require human-made prompts with the knowledge of the tasks at hand. We introduce a framework for what we call Combinatorial Reasoning (CR), a fully-automated prompting method, where reasons are sampled from an LLM pipeline and mapped into a Quadratic Unconstrained Binary Optimization (QUBO) problem. The framework investigates whether QUBO solutions can be profitably used to select a useful subset of the reasons to construct a Chain-of-Thought style prompt. We explore the acceleration of CR with specialized solvers. We also investigate the performance of simpler zero-shot strategies such as linear majority rule or random selection of reasons. Our preliminary study indicates that coupling a combinatorial solver to generative AI pipelines is an interesting avenue for AI reasoning and elucidates design principles for future CR methods.
Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization
Eindhoven University of Technology(埃因霍温理工大学);Brandenburg University of Technology(勃兰登堡工业大学);Nanyang Technological University(南洋理工大学);Delft University of Technology(代尔夫特理工大学);Southeast University(东南大学)
Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process
Beihang University, Beijing, China(北京航空航天大学);Zhongguancun Laboratory, Beijing, China(中关村实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Hong Kong University of Science and Technology(香港科学与技术大学);Xi'an Jiaotong University, Xi'an, China(西安交通大学);Tsinghua University, Beijing, China(清华大学);Singapore University of Technology and Design(新加坡科技设计大学)
Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challenges
Queensland University of Technology(昆士兰科技大学);Al-Ahliyya Amman University(安曼阿赫利亚大学);Budapest University of Technology and Economics(布达佩斯技术与经济大学);Arab American University(阿拉伯美国大学);Columbia University(哥伦比亚大学);Minia University(米尼亚大学)
CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Czech Technical University in Prague(捷克布拉格理工大学);CRRC Zhuzhou Institute(中车株洲研究所);Tengen Intelligence Institute(天元智能研究院);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE(教育部知识驱动人机智能工程研究中心)
Beyond Fine-Tuning: In-Context Learning and Chain-of-Thought for Reasoned Distractor Generation
School of Computing, Macquarie University, Australia(麦考瑞大学计算机学院,澳大利亚);School of Computer and Mathematical Sciences, Adelaide University, Australia(阿德莱德大学计算机与数学科学学院,澳大利亚);College of Engineering and Computing in Al-Lith, Umm Al-Qura University, Saudi Arabia(乌姆·阿尔·库拉大学阿尔·利思工程与计算学院,沙特阿拉伯)
Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
Alibaba Group(阿里巴巴集团);Beijing Jiaotong University(北京交通大学);Hong Kong University of Science and Technology(香港理工大学);Nanjing University(南京大学);Peking University(北京大学);OpenRLHF;CleanRL;Mila
Mixture of Demonstrations for Textual Graph Understanding and Question Answering
Independent Researcher, Wayne State University(韦恩州立大学独立研究者);Institute for Clarity in Documentation(文档清晰研究所);Inria Paris-Rocquencourt(巴黎-罗克琴court研究所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒尔研究实验室)
Compositional Language Understanding with Text-based Relational Reasoning
Mila(米拉研究所);McGill University(麦吉尔大学);Université de Montréal(蒙特利尔大学);Facebook AI Research (FAIR)(脸书人工智能研究院);Samsung Advanced Institute of Technology (SAIT)(三星综合技术院)