Resource-efficient training optimization techniques are becoming increasingly important as the size of large language models (LLMs) continues to grow. In particular, batch packing is commonly used in pre-training and supervised fine-tuning to achieve resource-efficient training. We propose preference packing, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO). Preference packing improves resource efficiency by reducing the attention operations for duplicate input prompts and decreasing KV cache memory usage. We conducted experiments on text-only datasets and image-included datasets and achieved at least 37% reduction in training time. Notably, this method can be applied alongside existing optimization techniques such as batch sorting, resulting in a 3.22x speedup.
Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
AI Technology Center of OVB, Tencent(腾讯OVB人工智能技术中心);Platform and Content Group, Tencent(腾讯平台与内容事业群);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院)
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
Graduate School of AI, KAIST(韩国科学技术院人工智能研究生院);Samsung Advanced Institute of Technology(三星先进技术研究院);Department of Electrical and Computer Engineering, UBC(不列颠哥伦比亚大学电气与计算机工程系)
Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
Shandong University(山东大学);Southern University of Science and Technology(南方科技大学);University of Georgia(佐治亚大学);National University of Singapore(新加坡国立大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Multi-Response Preference Optimization with Augmented Ranking Dataset
INMED DATA;Yonsei University(延世大学);Asan Medical Center(峨山医院);University of Ulsan College of Medicine(蔚山大学医学院);Asan Institute for Life Sciences(峨山生命科学研究所)
Multi-Objective Exploration and Preference Optimization via Mutual Information
School of Computer, Beihang University(北京航空航天大学计算机学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(星辰AGI实验室,中国电信人工智能技术(北京)有限公司)
Adaptive Preference Optimization with Uncertainty-aware Utility Anchor
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI)
ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning
Northwestern Polytechnical University, School of Automation, Xi’an, China(西北工业大学自动化学院,西安,中国);Northwestern Polytechnical University, School of Cybersecurity, Xi’an, China(西北工业大学网络安全学院,西安,中国);Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou, China(浙大宁波理工学院滨海人工智能研究院,杭州,中国);Alibaba Group, Hangzhou, China(阿里巴巴集团,杭州,中国)
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University(宾夕法尼亚州立大学人工智能研究实验室);University of Chinese Academy of Sciences(中国科学院大学);Tencent AI Lab(腾讯人工智能实验室)
DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Shenzhen Loop Area Institute(深圳河套学院);Shanghai Research Institute for Intelligent Autonomous Systems(上海智能自主系统研究院)