先前研究发现,Transformer在学习回答潜在两跳问题(形式如“谁是Bob母亲的老板?”的问题)方面能力不稳定。受Transformer用于简单事实记忆的知识容量相关前期研究启发,我们通过研究Transformer学习两跳问答(two-hop QA)数据集的能力如何随模型规模缩放,探究该现象的成因。我们发现,容量缩放与泛化表现均支持以下假设:潜在两跳QA需要Transformer对每个事实学习两次,而采用思维链(chain of thought)的两跳QA则无需如此。我们还表明,通过设置合适的数据集参数,可将极小模型“困在”一个独立记忆两跳问题答案的状态中——即便它们若能学会通过函数组合来答题会表现得更好。我们的研究结果表明,容量缩放测量可补充现有可解释性方法,不过将其用于该目的仍存在一些挑战。
英文摘要:
Prior work has found that transformers have an inconsistent ability to learn to answer latent two-hop questions -- questions of the form "Who is Bob's mother's boss?" We study why this is the case by examining how transformers' capacity to learn datasets of two-hop questions and answers (two-hop QA) scales with their size, motivated by prior work on transformer knowledge capacity for simple factual memorization. We find that capacity scaling and generalization both support the hypothesis that latent two-hop QA requires transformers to learn each fact twice, while two-hop QA with chain of thought does not. We also show that with appropriate dataset parameters, it is possible to "trap" very small models in a regime where they memorize answers to two-hop questions independently, even though they would perform better if they could learn to answer them with function composition. Our findings show that measurement of capacity scaling can complement existing interpretability methods, though there are challenges in using it for this purpose.
Adapting Meta Knowledge Graph Information for Multi-Hop Reasoning over Few-Shot Relations
Tsinghua University(清华大学);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);KIRC, Institute for Artificial Intelligence, Tsinghua University(清华大学人工智能研究院知识图谱与推理中心(KIRC));Beijing National Research Center for Information Science and Technology(北京国家信息科学与技术研究中心)
Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
UNC Chapel Hill(北卡罗来纳大学教堂山分校)
多跳QA需要一个模型来连接分散在长上下文中的多个证据来回答问题。最近提出的HotpotQA(Yang等人,2018)数据集由体现四种不同的多跳推理范例(两个桥接实体设置、检查多个属性和比较两个实体)的问题组成,使得单个神经网络处理所有四个问题具有挑战性。在这项工作中,我们提出了一个可解释的,基于神经网络的自组装神经模块网络(Hu et al.,2017年,2018年)的多跳推理,在那里我们设计了四个新的模块(查找,重新定位,比较,NoOp)来执行独特类型的语言推理。基于一个问题,我们的布局控制器RNN动态地推断一系列推理模块来构建整个网络。从经验上讲,我们表明,我们的动态,多跳模块化网络实现了显着的改进,静态,单跳基线(定期和对抗性评估)。我们通过三个分析进一步证明了我们的模型的可解释性。首先,控制器可以将多跳问题软分解为多个单跳子问题,以促进主网络的组合推理行为。其次,控制器可以预测符合人类专家设计的布局的布局。最后,中间模块可以通过处理来自控制器的子问题来推断连接两个远距离定位的支持事实的实体。
Examining the Vulnerability of Multi-Agent Medical Systems to Human Interventions for Clinical Reasoning
Stanford University(斯坦福大学);Jordan High School(乔丹高中);UC San Diego(加利福尼亚大学圣迭戈分校);Winchester High School(温彻斯特高中);James Logan High School(詹姆斯·洛根高中);Rutgers University(罗格斯大学);Foothill–De Anza College(福希尔-德安扎学院);Fordham University(福特汉姆大学);Chapman University(查普曼大学);UC Berkeley(加利福尼亚大学伯克利分校)
DTKG: Dual-Track Knowledge Graph-Verified Reasoning Framework for Multi-Hop QA
School of Computer Science;Engineering, Beihang University, Beijing, China;School of Reliability;Systems Engineering, Beihang University, Beijing, China;State Key Laboratory of Complex \& Critical Software Environment;National Key Laboratory of Reliability;State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences
CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps
Department of Artificial Intelligence, Chung-Ang University(Chung-Ang大学人工智能系);Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University(Chung-Ang大学高级成像科学研究院,多媒体与电影)
Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?
ASUS Intelligent Cloud Services (AICS)(华硕智能云服务(AICS));National University of Singapore(新加坡国立大学);University of Manchester(曼彻斯特大学);Imperial Global Singapore(帝国理工学院新加坡分部)
CryptoFlowBench: Benchmarking Multi-Hop Reasoning on Blockchains for Cryptocurrency Fund Tracing
State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);Hangzhou City University(杭州城市学院);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术开发区(滨江)区块链与数据安全研究院);China Mobile Group Zhejiang Co., Ltd(中国移动通信集团浙江有限公司);China Unicom Digital Technology Co., Ltd(中国联通数字科技有限公司)
Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models
Chandar Research Lab(Chandar研究实验室);Mila – Quebec AI Institute(魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Microsoft Research(微软研究院);Polytechnique Montréal(蒙特利尔理工学院);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
THOR: A Theta-Gamma Hierarchical Oscillatory Reasoning Framework for Multi-hop QA
Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州高等研究院);School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China(中国科学技术大学生命科学与医学部生物医学工程学院)
GMH: A General Multi-hop Reasoning Model for KG Completion
Nankai University(南开大学);Sichuan University(四川大学);University of Innsbruck(因斯布鲁克大学);National University of Singapore(新加坡国立大学);Mashang Consumer Finance Co., Ltd.(马上消费金融股份有限公司)
Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models
State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);University of Science and Technology of China(中国科学技术大学);The Hong Kong University of Science and Technology(香港科技大学)
From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
Faculty of Computer Science(计算机科学系);UniVie Doctoral School Computer Science(维也纳大学计算机科学博士学院);Faculty of Philological and Cultural Studies(文学与文化研究系);University of Vienna(维也纳大学);Saarland University(萨尔兰大学);Eastern Institute of Technology(东部技术研究所);Department of Statistics(统计系);LMU Munich(慕尼黑莱茵河大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)