arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.01482cs.AI
收藏

通过提示方法与模型对答案的解释理解 LLM 的科学推理

Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers

University of Toronto Department of Psychiatry(多伦多大学精神病学系);Toronto Metropolitan University(多伦多 Metropolitan 大学);St. Michael’s Hospital, Unity Health Toronto(圣米歇尔医院,统一健康多伦多);Department of Electrical, Computer, and Biomedical Engineering(电气、计算机和生物医学工程系);University of Waterloo(滑铁卢大学);Department of Management Science and Engineering(管理科学与工程系)

Alice Rueda, Mohammed S. Hassan, Argyrios Perivolaris, Bazen G. Teferra, Reza Samavi, Sirisha Rambhatla, Yuqi Wu, Yanbo Zhang, Bo Cao, Divya Sharma, Sridhar Krishnan, Venkat Bhat

AI总结:

本文在 GPQA 上比较 GPT-4o 的七种提示策略,发现简单方法科学推理表现较好,并提出结合结构化推理、混合 AI 与人在回路的改进方向。

摘要:

大语言模型(LLM)已在多个领域的自然语言理解、推理和问题解决中展现出卓越能力。然而,它们执行复杂、多步推理任务的能力——这对科学、医学和法律领域的应用至关重要——仍是一个活跃研究方向。本文考察当代 LLM 的推理能力,分析其优势、局限和改进潜力。研究在 Graduate-Level GoogleProof Q&A(GPQA)数据集上使用提示工程技术评估 GPT-4o 的科学推理能力。实验测试了五种流行提示工程技术和两种定制提示:基线直接回答(零样本)、思维链(CoT)、零样本 CoT、自我提问、自洽性、分解以及多路径提示。研究发现,尽管 LLM 表现出涌现推理能力,但它们往往依赖模式识别而非真正的逻辑推断,从而导致复杂问题解决中的不一致。结果表明,自洽性优于其他提示工程技术,准确率为 52.99%,其次是直接回答(52.23%)。零样本 CoT(50%)优于多路径(48.44%)、分解(47.77%)、自我提问(46.88%)和 CoT(43.75%)。自洽性在解释答案方面表现倒数第二。直接回答、CoT 和零样本 CoT 等简单技术具有最佳的科学推理表现。我们提出一项研究议程,旨在通过整合结构化推理框架、混合 AI 方法和人在回路方法来弥合这些差距。通过批判性评估 LLM 的推理机制,本文为关于通用人工智能未来以及更稳健、可信 AI 系统发展的持续讨论作出贡献。

英文摘要:

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding, reasoning, and problem-solving across various domains. However, their ability to perform complex, multi-step reasoning task-essential for applications in science, medicine, and law-remains an area of active investigation. This paper examines the reasoning capabilities of contemporary LLMs, analyzing their strengths, limitations, and potential for improvement. The study uses prompt engineering techniques on the Graduate-Level GoogleProof Q&A (GPQA) dataset to assess the scientific reasoning of GPT-4o. Five popular prompt engineering techniques and two tailored promptings were tested: baseline direct answer (zero-shot), chain-of-thought (CoT), zero-shot CoT, self-ask, self-consistency, decomposition, and multipath promptings. Our findings indicate that while LLMs exhibit emergent reasoning abilities, they often rely on pattern recognition rather than true logical inference, leading to inconsistencies in complex problem-solving. The results indicated that self-consistency outperformed the other prompt engineering technique with an accuracy of 52.99%, followed by direct answer (52.23%). Zero-shot CoT (50%) outperformed multipath (48.44%), decomposition (47.77%), self-ask (46.88%), and CoT (43.75%). Self-consistency performed the second worst in explaining the answers. Simple techniques such as direct answer, CoT, and zero-shot CoT have the best scientific reasoning. We propose a research agenda aimed at bridging these gaps by integrating structured reasoning frameworks, hybrid AI approaches, and human-in-the-loop methodologies. By critically evaluating the reasoning mechanisms of LLMs, this paper contributes to the ongoing discourse on the future of artificial general intelligence and the development of more robust, trustworthy AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑