arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.22735cs.CL
收藏

面向矛盾调和的大语言模型解释生成

Explanation Generation for Contradiction Reconciliation with LLMs

University of Sheffield, UK(谢菲尔德大学)

Jason Chan, Zhixue Zhao, Robert Gaizauskas

AI总结:

提出矛盾调和解释生成任务,通过改造NLI数据集和设计质量指标,评估18个LLM在该任务上的表现,发现模型能力有限且增大模型规模时“思考”收益递减。

摘要:

现有的NLP工作通常将矛盾视为需要通过选择接受或拒绝哪些陈述来解决的错误。然而,在社交互动和专业领域中,人类推理的一个关键方面是能够假设调和矛盾的解释。例如,“Cassie讨厌咖啡”和“她每天买咖啡”看似矛盾,但如果Cassie有每天为所有同事买咖啡这一不令人羡慕的日常任务,那么两者是兼容的。尽管大语言模型(LLM)的推理能力不断增强,但它们假设这种调和解释的能力在很大程度上仍未探索。为了填补这一空白,我们引入了调和解释生成任务,其中模型必须生成能够有效使矛盾陈述兼容的解释。我们提出了一种改造现有自然语言推理(NLI)数据集的新方法,并引入了可实现可扩展自动评估的质量指标。对18个LLM的实验表明,大多数模型在此任务中取得的成功有限,并且通过“思考”延长测试时计算的好处随着模型规模的增大而趋于平稳。我们的结果突显了LLM推理中一个未被充分探索的维度,以及解决这一限制以增强LLM下游应用(如聊天机器人和科学助手)的必要性。

英文摘要:

Existing NLP work commonly treats contradictions as errors to be resolved by choosing which statements to accept or discard. Yet a key aspect of human reasoning in social and professional settings is the ability to hypothesize explanations that reconcile contradictions. For example, "Cassie hates coffee" and "She buys coffee everyday" appear contradictory, but are in fact compatible if Cassie was assigned the daily chore of buying coffee for all her coworkers. Despite advances in the reasoning capabilities of large language models (LLMs), their ability to hypothesize such explanations remains largely unexplored. To address this gap, we introduce the task of reconciliatory explanation generation, where models must generate explanations that effectively render contradictory statements compatible. We propose a novel method of reusing natural language inference (NLI) datasets, and introduce quality metrics that enable scalable automatic evaluation. Experiments with 13 LLMs in thinking and non-thinking modes reveal models' limited success in this task, with longer reasoning chains negatively associated with explanation success. As a proof of concept, we further adapt our introduced task and metrics for reinforcement-learning training, finding that trained models exhibit improved accuracy in non-thinking mode on two narrative reasoning benchmarks (True Detective and MuSR). Our findings highlight reconciliatory explanation generation as a challenge in LLM reasoning and motivate further work to investigate its potential for improving model capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑