Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work.
Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Electronic Science Research Institute of China Electronics(中国电子科技集团电子科学研究院)
CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China(上海交通大学计算机科学学院);Shanghai KEY Laboratory of Integrated Administration Technologies for Information Security, Shanghai, China(上海信息安全集成管理技术重点实验室);School of Politics, National Defense University, Shanghai, China(国防大学政治学院)
Intercepting the Kangaroo: Experimental Astrolinguistics with Constructed Lexicons, Active Probing, and Large Language Models as Informants and Hypothesis Proposers
Generalized Category Discovery with Large Language Models in the Loop
School of Automation Science and Engineering, Xi’an Jiaotong University(西安交通大学自动化科学与工程学院);School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Lenovo Research(联想研究院);University of Massachusetts Boston(马萨诸塞大学波士顿分校)
Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning
Key Laboratory of Big Data & Artificial Intelligence in Transportation, Beijing Jiaotong University(大数据与人工智能在交通中的关键实验室,北京交通大学);School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学);Pattern Recognition Center, WeChat AI, Tencent Inc(模式识别中心,微信AI,腾讯公司)