arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.04086cs.CL
收藏

利用弱监督优化语言模型的推理能力

Optimizing Language Model's Reasoning Abilities with Weak Supervision

University of California, San Diego(加利福尼亚大学圣迭戈分校);University of Southern California(南加利福尼亚大学);University of Pennsylvania(宾夕法尼亚大学);Yale University(耶鲁大学);Washington University in St. Louis(圣路易斯华盛顿大学)

Yongqi Tong, Sizhe Wang, Dawei Li, Yifan Wang, Simeng Han, Zi Lin, Chengsong Huang, Jiaxin Huang, Jingbo Shang

AI总结:

本文提出自强化方法,仅用少量标注问题微调模型,并通过未标注数据上模型响应差异迭代提升推理能力,同时构建了含25,147个问题及10,000个未标注问题的弱监督基准PuzzleBen,验证了方法的有效性。

摘要:

尽管大型语言模型(LLMs)已展现出处理复杂查询的能力,但以往大量工作依赖于人类专家大规模标注的数据集。然而,这种对全监督标注的依赖带来了可扩展性挑战,尤其是在模型和数据需求不断增长的情况下。为缓解这一问题,我们探索了在极少人类监督下增强LLMs推理能力的潜力。在本工作中,我们提出了自强化(self-reinforcement)方法,该方法首先使用少量标注问题对模型进行监督微调(Supervised Fine-Tuning,SFT),随后通过从未标注问题上SFT模型与未微调模型响应之间的差异中学习,迭代地改进LLMs。我们的方法提供了一种高效的途径,无需严重依赖大量人工标注的解释。然而,当前的推理基准通常只包含黄金参考答案或推理过程。因此,我们提出了PuzzleBen,一个弱监督基准,包含25,147个复杂问题、答案以及人类生成的推理过程,涵盖脑筋急转弯、谜题、谜语、乱序段落和批判性推理等多种领域任务。我们数据集的一个独特之处在于包含了10,000个未标注问题,使我们能够探索利用更少的超大规模数据来提升LLMs的推理能力。我们的实验强调了PuzzleBen的重要性,以及我们方法作为未来工作中一个有前景方向的有效性。我们的数据集和代码将很快在Anonymity Link上发布。

英文摘要:

While Large Language Models (LLMs) have demonstrated proficiency in handling complex queries, much of the past work has depended on extensively annotated datasets by human experts. However, this reliance on fully-supervised annotations poses scalability challenges, particularly as models and data requirements grow. To mitigate this, we explore the potential of enhancing LLMs' reasoning abilities with minimal human supervision. In this work, we introduce self-reinforcement, which begins with Supervised Fine-Tuning (SFT) of the model using a small collection of annotated questions. Then it iteratively improves LLMs by learning from the differences in responses from the SFT and unfinetuned models on unlabeled questions. Our approach provides an efficient approach without relying heavily on extensive human-annotated explanations. However, current reasoning benchmarks typically only include golden-reference answers or rationales. Therefore, we present \textsc{PuzzleBen}, a weakly supervised benchmark that comprises 25,147 complex questions, answers, and human-generated rationales across various domains, such as brainteasers, puzzles, riddles, parajumbles, and critical reasoning tasks. A unique aspect of our dataset is the inclusion of 10,000 unannotated questions, enabling us to explore utilizing fewer supersized data to boost LLMs' inference capabilities. Our experiments underscore the significance of \textsc{PuzzleBen}, as well as the effectiveness of our methodology as a promising direction in future endeavors. Our dataset and code will be published soon on \texttt{Anonymity Link}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑