Direct Alignment of Language Models via Quality-Aware Self-Refinement
Hong Kong University of Science and Technology(香港科技大学);LightSpeed Studios, Tencent(腾讯光速工作室);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, CAS(中国科学院香港创新研究院人工智能与机器人中心)
Runsheng Yu, Yong Wang, Xiaoqi Jiao, Youzhi Zhang, James T. Kwok
Reinforcement Learning from Human Feedback (RLHF) has been commonly used to align the behaviors of Large Language Models (LLMs) with human preferences. Recently, a popular alternative is Direct Policy Optimization (DPO), which replaces an LLM-based reward model with the policy itself, thus obviating the need for extra memory and training time to learn the reward model. However, DPO does not consider the relative qualities of the positive and negative responses, and can lead to sub-optimal training outcomes. To alleviate this problem, we investigate the use of intrinsic knowledge within the on-the-fly fine-tuning LLM to obtain relative qualities and help to refine the loss function. Specifically, we leverage the knowledge of the LLM to design a refinement function to estimate the quality of both the positive and negative responses. We show that the constructed refinement function can help self-refine the loss function under mild assumptions. The refinement function is integrated into DPO and its variant Identity Policy Optimization (IPO). Experiments across various evaluators indicate that they can improve the performance of the fine-tuned models over DPO and IPO.
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
Soochow University(苏州大学);Zhipu AI(智谱AI);Tsinghua University(清华大学);Key Laboratory of Data Intelligence and Advanced Computing, Soochow University(苏州大学数据智能与先进计算重点实验室)
Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning
Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);Huawei Noah’s Ark Lab(华为诺亚实验室);College of AI, Tsinghua University(清华大学人工智能学院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(计算与智能研究所,哈尔滨工业大学,深圳,中国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高科技园区(滨江)区块链与数据安全研究院);Sun Yat-sen University(中山大学);KAUST(卡塔尔大学)
MAR: Efficient Large Language Models via Module-aware Architecture Refinement
Department of CSE, Southern University of Science and Technology(南方科技大学计算机科学与工程系);School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);National Engineering Laboratory for Big Data Analytics, Xi’an Jiaotong University(西安交通大学大数据分析国家工程实验室);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);ACS Laboratory, Huawei Technologies Co., Ltd.(华为技术有限公司ACS实验室)
3D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinement
College of Computer Science, Beihang University(北京航空航天大学计算机科学学院);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);Department of Information Engineering and Computer Science, University of Trento(特伦托大学信息工程与计算机科学系);Meituan(美团)
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement
Department of Computer Science and Engineering, Dongguk University(计算机科学与工程系,东国大学);Department of Computer Science and Artificial Intelligence, Dongguk University(计算机科学与人工智能系,东国大学)
AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing
Department of Mathematics and Statistics(数学与统计学系);Georgia State University(佐治亚州立大学);Department of Probability and Statistics(概率与统计学系);Department of Computer Science and Engineering(计算机科学与工程系);Michigan State University(密歇根州立大学);Concordia University(Concordia 大学);Department of Statistics(统计学系);University of Manitoba(曼尼托巴大学)
SEG:Seeds-Enhanced Iterative Refinement Graph Neural Network for Entity Alignment
College of Computer;Mathematics Central South University of Forestry;4em College of Computer;Mathematics 4em Central South University of Forestry;3em Department of Computer Science 3em State University of New York 3em New Paltz, New York 12561, USA 4em
Heterogeneous Element-Aware Cross-Version Differencing of Scientific Documents via Layout-Aware Alignment and Structure-Aware Reasoning
Beijing Renhe Information Technology Co., Ltd.(北京人和信息技术有限公司);Key Laboratory of Digital Publishing and Total Process Management of Scientific and Technical Journals(科技期刊数字出版与全程管理重点实验室)
RAGA: Relation-aware Graph Attention Networks for Global Entity Alignment
School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);National Engineering Research Center for Software Engineering, Peking University(北京大学国家软件工程研究中心);Key Laboratory of High Confidence Software Technologies (PKU), Ministry of Education(教育部高可信软件技术重点实验室(北京大学))
EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院)