Answer verification identifies correct solutions among candidates generated by large language models (LLMs). Current approaches typically train verifier models by labeling solutions as correct or incorrect based solely on whether the final answer matches the gold answer. However, this approach neglects any flawed rationale in the solution yielding the correct answer, undermining the verifier's ability to distinguish between sound and flawed rationales. We empirically show that in StrategyQA, only 19% of LLM-generated solutions with correct answers have valid rationales, thus leading to an unreliable verifier. Furthermore, we demonstrate that training a verifier on valid rationales significantly improves its ability to distinguish valid and flawed rationale. To make a better verifier without extra human supervision, we introduce REPS (Rationale Enhancement through Pairwise Selection), a method for selecting valid rationales from candidates by iteratively applying pairwise self-evaluation using the same LLM that generates the solutions. Verifiers trained on solutions selected by REPS outperform those trained using conventional training methods on three reasoning benchmarks (ARC-Challenge, DROP, and StrategyQA). Our results suggest that training reliable verifiers requires ensuring the validity of rationales in addition to the correctness of the final answers, which would be critical for models assisting humans in solving complex reasoning tasks.
CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China(上海交通大学计算机科学学院);Shanghai KEY Laboratory of Integrated Administration Technologies for Information Security, Shanghai, China(上海信息安全集成管理技术重点实验室);School of Politics, National Defense University, Shanghai, China(国防大学政治学院)
Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification
University of Warwick(沃里克大学);Queen Mary University of London(伦敦大学女王学院);University of Bologna(博洛尼亚大学);King’s College London(伦敦大学国王学院);The Alan Turing Institute(艾伦·图灵研究所)
A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution
School of Engineering, Westlake University(西湖大学工程学院);Zhejiang University(浙江大学);Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究院)
针对跨文档事件共指消解中触发词词汇匹配的虚假偏差,本文基于结构因果模型提出 LLM 参与闭环的理据中心反事实数据增强方法,在三个基准上达到 SOTA 并具备域外鲁棒性。
Candidate Generation and Definition-Guided Verification for Sentence-Level Depression Symptom Recognition
Institute for Systems and Robotics (ISR)(系统与机器人研究所);Instituto Superior Técnico (IST)(高等技术学院);University of Lisbon(里斯本大学);Hospital Beatriz Ângelo(Beatriz Ângelo医院)
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
Shanghai AI Laboratory(上海人工智能实验室);MMLab, The Chinese University of Hong Kong(香港中文大学多媒体实验室);Shanghai Jiao Tong University(上海交通大学);Great Bay University(大湾区大学)
Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thought
Center for Advanced Intelligence Project, RIKEN(理化学研究所先进智能项目中心);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院)
Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);Alibaba Group(阿里巴巴集团);Renmin University of China(中国人民大学);Xiaohongshu(小红书);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Shanghai Jiaotong University(上海交通大学);Peking University(北京大学)
VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
The Hong Kong University of Science and Technology(香港科学与技术大学);Nanyang Technological University(南洋理工大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence
School of Computing, National University of Singapore(新加坡国立大学计算学院);Xiaohongshu Inc.(小红书公司);School of Cyber Science and Technology, Zhejiang University(浙江大学网络空间安全学院);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);School of Statistics and Data Science, Shanghai University of Finance and Economics(上海财经大学统计与数据科学学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)
EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Taobao & Tmall Group(淘宝天猫集团);Shandong University(山东大学);Peking University(北京大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Reinforcing Step-level Reasoning for Effective Self-Correction in LLMs
Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);VinUniversity;Institute for Infocomm Research (IR), A*STAR(新加坡科技研究局信息通信研究院)
TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
Mohamed bin Zayed University of Artificial Intelligence(莫莫德·本·扎耶德人工智能大学);Monash University(墨尔本大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);University of Southampton(南安普顿大学)
Are Rationales Necessary and Sufficient? Tuning LLMs for Explainable Misinformation Detection
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Tongyi Lab, Alibaba Group(阿里云实验室);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)