arXiv 2310.094302025-01-17cs.CLcs.AI

评估并增强大语言模型在逻辑推理任务中面对任务结构变化的鲁棒性

Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning

Strong AI Lab, NAOInstitute, Waipapa Taumata Rau - The University of Auckland（强人工智能实验室、NAO研究所、怀帕帕·陶马陶·劳 - 奥克兰大学）；Xtracta, New Zealand（Xtracta公司）；School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）

Qiming Bao, Gael Gendron, Alex Yuxuan Peng, Wanjun Zhong, Neset Tan, Yang Chen, Michael Witbrock, Jiamou Liu

备注：: The short version (v3) was accepted for oral presentation at the first LLM@IJCAI 2023 non-archival symposium, and the full version was accepted by ICONIP 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究针对大语言模型逻辑推理鲁棒性评估不足的问题，构建了三个带任务结构变体的新逻辑推理数据集，发现现有模型鲁棒性较差，并提出通过训练集引入任务变体、逻辑驱动数据增强的方法提升其泛化与鲁棒性。

摘要：

大语言模型（LLM），例如LLaMA、Alpaca、Vicuna、GPT-3.5和GPT-4，已将AI系统在各类自然语言处理任务上的性能提升至类人水平。然而，它们在执行逻辑推理时的泛化能力与鲁棒性尚未得到充分评估。为全面评估这一能力，我们构建了三个全新的逻辑推理数据集，分别命名为ReClor-plus、LogiQA-plus和LogiQAv2-plus，它们对标准逻辑推理数据集进行了扩展，用于评估LLM推理的鲁棒性。针对每个数据集，我们构建了三个子集：第一个子集的选项被随机打乱，第二个子集的正确选项被替换为“其他选项均不正确”，第三个子集结合了打乱与替换操作。在这些数据集上开展的实验表明，这些简单的增强操作极大地阻碍了模型的性能。尽管所有模型在原始公开数据集上表现优异，但我们发现它们在这些新构建的数据集上表现均很差。我们还证明，在训练集中引入任务变体能够显著提升模型在原始数据集和我们构建的数据集上的性能。最后，我们表明，应用逻辑驱动的数据增强进行微调与提示工程，能够提升判别式模型和生成式模型的泛化能力，为提高它们在涉及逻辑推理的任务中的鲁棒性提供了一条路径。源代码和数据已在https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning 公开提供。

英文摘要：

Large language models (LLMs), such as LLaMA, Alpaca, Vicuna, GPT-3.5 and GPT-4, have advanced the performance of AI systems on various natural language processing tasks to human-like levels. However, their generalisation and robustness when performing logical reasoning has not been sufficiently assessed. To comprehensively evaluate this ability, we develop three new logical reasoning datasets named "ReClor-plus", "LogiQA-plus" and "LogiQAv2-plus" that extend standard logical reasoning datasets to evaluate the robustness of the LLM's reasoning. For each, we create three subsets: the first with randomly shuffled options, the second with the correct choices replaced by "none of the other options is correct", and the third with a combination of shuffling and substitution. Experiments on these datasets show that these simple augmentations greatly hinder the models' performance. Despite their high performance on the original publicly available datasets, we find that all models perform poorly on these newly constructed datasets. We also demonstrate that introducing task variations into the training set can markedly improve the model's performance on both the original and our developed datasets. Finally, we show that applying logic-driven data augmentation for fine-tuning and prompting can enhance generalisation in both discriminative and generative models, offering a path to improving their robustness for tasks involving logical reasoning. Source code and data are made publicly available at https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。