arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.02567cs.CLcs.AI
收藏

通过代码引导大语言模型实现更优的多语言结构化推理

Eliciting Better Multilingual Structured Reasoning from LLMs through Code

University of Pennsylvania(宾夕法尼亚大学);AI Labs(AI实验室)

Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, Saab Mansour

AI总结:

本文提出xSTREET多语言推理数据集,并通过代码注释增强训练和代码原语提示推理两种方法,有效缩小了LLM在非英语推理任务上的性能差距。

摘要:

大语言模型(LLM)的发展在推理方面已展现出进展,但现有研究大多仅考虑英语或简单推理任务。为解决这一问题,我们引入了一个多语言结构化推理与解释数据集,命名为xSTREET,涵盖六种语言上的四项任务。xSTREET揭示了基础LLM在英语与非英语推理任务之间的性能差距。随后,我们基于“在代码上训练的LLM是更优秀的推理器”这一洞见,提出两种方法来弥补该差距。第一,在训练阶段,我们利用机器翻译为代码数据集增加多语言注释,同时保持程序代码原样不变。第二,在推理阶段,我们通过采用一种提示结构来弥合训练与推理之间的差距,该结构结合逐步的代码原语以推导新事实并找到解决方案。我们的方法在xSTREET上展现出提升的多语言性能,尤其是在科学常识推理子任务上表现最为显著。此外,模型在非推理任务上没有出现性能退化,从而证明我们的技术保持了通用能力。

英文摘要:

The development of large language models (LLM) has shown progress on reasoning, though studies have largely considered either English or simple reasoning tasks. To address this, we introduce a multilingual structured reasoning and explanation dataset, termed xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks. We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multilingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus demonstrating our techniques maintain general-purpose abilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑