arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.11833cs.CLcs.AI
收藏

你的大语言模型是否陷入思维定势?关于思维定势如何影响大语言模型推理能力的调查研究

Is your LLM trapped in a Mental Set? Investigative study on how mental sets affect the reasoning capabilities of LLMs

IIT Bombay(印度理工学院孟买分校);Hyperbots Inc(Hyperbots公司)

Saiful Haq, Niyati Chhaya, Piyush Pandey, Pushpak Bhattacharya

AI总结:

本研究首次将认知心理学中的思维定势概念引入大语言模型评估,比较Llama-3.1和GPT-4o在复杂推理任务中的表现,揭示现有基准忽视适应性维度的问题。

摘要:

在本文中,我们提出了一项关于思维定势如何影响大语言模型推理能力的调查研究。得益于参数高效微调(PEFT)和情境学习(ICL)等涌现能力的进步,大语言模型在各种自然语言处理(NLP)任务中表现出色。对于复杂的推理任务,选择合适的模型进行PEFT或ICL至关重要,这通常依赖于MMLU、MATH和GSM8K等基准测试的得分。然而,当前的评估方法基于F1分数或更大模型的推理链评估等指标,忽略了一个关键维度:对陌生情境的适应能力和克服根深蒂固的思维模式的能力。在认知心理学中,思维定势指的是即使先前成功的策略变得低效时仍倾向于坚持使用,这对问题解决和推理构成挑战。我们比较了Llama-3.1-8B-Instruct、Llama-3.1-70B-Instruct和GPT-4o等大语言模型在思维定势存在时的表现。据我们所知,这是第一项将认知心理学概念整合到复杂推理任务的大语言模型评估中的研究,为它们的适应性和问题解决效能提供了更深入的见解。

英文摘要:

In this paper, we present an investigative study on how Mental Sets influence the reasoning capabilities of LLMs. LLMs have excelled in diverse natural language processing (NLP) tasks, driven by advancements in parameter-efficient fine-tuning (PEFT) and emergent capabilities like in-context learning (ICL). For complex reasoning tasks, selecting the right model for PEFT or ICL is critical, often relying on scores on benchmarks such as MMLU, MATH, and GSM8K. However, current evaluation methods, based on metrics like F1 Score or reasoning chain assessments by larger models, overlook a key dimension: adaptability to unfamiliar situations and overcoming entrenched thinking patterns. In cognitive psychology, Mental Set refers to the tendency to persist with previously successful strategies, even when they become inefficient - a challenge for problem solving and reasoning. We compare the performance of LLM models like Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct and GPT-4o in the presence of mental sets. To the best of our knowledge, this is the first study to integrate cognitive psychology concepts into the evaluation of LLMs for complex reasoning tasks, providing deeper insights into their adaptability and problem-solving efficacy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑