Large language models (LLMs) have shown remarkable capability in numerous tasks and applications. However, fine-tuning LLMs using high-quality datasets under external supervision remains prohibitively expensive. In response, LLM self-improvement approaches have been vibrantly developed recently. The typical paradigm of LLM self-improvement involves training LLM on self-generated data, part of which may be detrimental and should be filtered out due to the unstable data quality. While current works primarily employs filtering strategies based on answer correctness, in this paper, we demonstrate that filtering out correct but with high distribution shift extent (DSE) samples could also benefit the results of self-improvement. Given that the actual sample distribution is usually inaccessible, we propose a new metric called DS weight to approximate DSE, inspired by the Importance Weighting methods. Consequently, we integrate DS weight with self-consistency to comprehensively filter the self-generated samples and fine-tune the language model. Experiments show that with only a tiny valid set (up to 5\% size of the training set) to compute DS weight, our approach can notably promote the reasoning ability of current LLM self-improvement methods. The resulting performance is on par with methods that rely on external supervision from pre-trained reward models.
Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);Alibaba Group(阿里巴巴集团);Renmin University of China(中国人民大学);Xiaohongshu(小红书);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Shanghai Jiaotong University(上海交通大学);Peking University(北京大学)
Token-Importance Guided Direct Preference Optimization
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));ByteDance(字节跳动);Microsoft Research AI4Science(微软研究院AI4Science);University of Science and Technology Beijing(北京科技大学)
Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);LLM Team, Shopee Pte. Ltd.(Shopee Pte. Ltd. 语言模型团队);Beijing Key Laboratory of Research on Large Models and Intelligent Governance and Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(北京大型模型与智能治理研究重点实验室)
Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting
TUM School of Computation, Information and Technology(慕尼黑技术大学计算、信息与技术学院);Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Department of Computing, Imperial College London(伦敦帝国学院计算机系)