Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers
University of Toronto Department of Psychiatry(多伦多大学精神病学系);Toronto Metropolitan University(多伦多 Metropolitan 大学);St. Michael’s Hospital, Unity Health Toronto(圣米歇尔医院,统一健康多伦多);Department of Electrical, Computer, and Biomedical Engineering(电气、计算机和生物医学工程系);University of Waterloo(滑铁卢大学);Department of Management Science and Engineering(管理科学与工程系)
Alice Rueda, Mohammed S. Hassan, Argyrios Perivolaris, Bazen G. Teferra, Reza Samavi, Sirisha Rambhatla, Yuqi Wu, Yanbo Zhang, Bo Cao, Divya Sharma, Sridhar Krishnan, Venkat Bhat
Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding, reasoning, and problem-solving across various domains. However, their ability to perform complex, multi-step reasoning task-essential for applications in science, medicine, and law-remains an area of active investigation. This paper examines the reasoning capabilities of contemporary LLMs, analyzing their strengths, limitations, and potential for improvement. The study uses prompt engineering techniques on the Graduate-Level GoogleProof Q&A (GPQA) dataset to assess the scientific reasoning of GPT-4o. Five popular prompt engineering techniques and two tailored promptings were tested: baseline direct answer (zero-shot), chain-of-thought (CoT), zero-shot CoT, self-ask, self-consistency, decomposition, and multipath promptings. Our findings indicate that while LLMs exhibit emergent reasoning abilities, they often rely on pattern recognition rather than true logical inference, leading to inconsistencies in complex problem-solving. The results indicated that self-consistency outperformed the other prompt engineering technique with an accuracy of 52.99%, followed by direct answer (52.23%). Zero-shot CoT (50%) outperformed multipath (48.44%), decomposition (47.77%), self-ask (46.88%), and CoT (43.75%). Self-consistency performed the second worst in explaining the answers. Simple techniques such as direct answer, CoT, and zero-shot CoT have the best scientific reasoning. We propose a research agenda aimed at bridging these gaps by integrating structured reasoning frameworks, hybrid AI approaches, and human-in-the-loop methodologies. By critically evaluating the reasoning mechanisms of LLMs, this paper contributes to the ongoing discourse on the future of artificial general intelligence and the development of more robust, trustworthy AI systems.
When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness
LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Imperial College London(伦敦帝国学院);Technical University of Munich(慕尼黑工业大学);MaiNLP lab, CIS, LMU Munich(慕尼黑大学CIS研究所MaiNLP实验室)
A Sentiment Consolidation Framework for Meta-Review Generation
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
Enhance Multi-domain Sentiment Analysis of Review Texts through Prompting Strategies
Hong Kong Baptist University(香港浸会大学);BNU-UIC Institute of AI and Future Networks(北师大-联合国际学院人工智能与未来网络研究院);BNU-HKBU United International College(北师大-港浸大联合国际学院);Beijing Normal University(北京师范大学)
Self-Prompting Large Language Models for Zero-Shot Open-Domain QA
Shanghai Jiao Tong University(上海交通大学);SJTU-Paris Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎卓越工程师学院);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院)
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
School of Engineering, Westlake University(西湖大学工程学院);City University of Hong Kong(香港城市大学);Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);Johns Hopkins University(约翰斯·霍普金斯大学)
AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));City University of Hong Kong(香港城市大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);The Hong Kong University of Science and Technology(香港科技大学);University of California, Merced(加州大学默塞德分校);Chongqing University(重庆大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)