本文提出免微调的 PANDA 偏好适配方法，通过借鉴专家模型的响应偏好增强 LLM 的领域能力，在分类、交互决策及 ScienceWorld 任务上取得显著提升。
摘要：
尽管大语言模型（LLM）已在各类自然语言任务上展现出相当强的能力，但其表现往往不及特定领域的最先进模型。提升 LLM 领域特定能力的一种可能方法是使用相应数据集对其进行微调。然而，这种方法既耗费资源又耗费时间，而且不适用于闭源商业 LLM。在本文中，我们提出了 Preference Adaptation for Enhancing Domain-specific Abilities of LLMs（PANDA，即用于增强 LLM 领域特定能力的偏好适配），该方法旨在通过利用专家模型响应偏好中的洞见来增强 LLM 的领域特定能力，而无需微调。实验结果表明，PANDA 在文本分类和交互式决策任务上显著提升了 LLM 的领域特定能力。此外，配备 PANDA 的 LLM 在 ScienceWorld 的 4 项任务上甚至优于其所学习的专家模型。这一发现凸显了探索免调优方法以实现弱到强泛化的潜力。
英文摘要：
While Large language models (LLMs) have demonstrated considerable capabilities across various natural language tasks, they often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. However, this method can be both resource and time-intensive, and not applicable to closed-source commercial LLMs. In this paper, we propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA), a method designed to augment the domain-specific capabilities of LLMs by leveraging insights from the response preference of expert models without requiring fine-tuning. Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks. Moreover, LLM with PANDA even outperforms the expert model that being learned on 4 tasks of ScienceWorld. This finding highlights the potential of exploring tuning-free approaches to achieve weak-to-strong generalization.
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；University of Chicago（芝加哥大学）；The Chinese University of Hong Kong（香港中文大学）；East China Normal University（华东师范大学）
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Pattern Recongnition Center, WechatAI, Tencent Inc（图案识别中心，微信AI，腾讯公司）；Dept. of Comp. Sci. & Tech., Tsinghua University（计算机科学与技术系，清华大学）
CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Xiaohongshu Inc（小红书公司）
MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学上海智能信息处理重点实验室）
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers
MIT-IBM Watson AI Lab（MIT-IBM Watson人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）；Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）；IBM Research（IBM研究院）；University of California, Berkeley（加州大学伯克利分校）
RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际 Graduate School）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Yinwang Intelligent Technology Co., Ltd.（亿纬智能科技有限公司）；Huawei’s 2012 Lab（华为2012实验室）
Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Georgia Institute of Technology（佐治亚理工学院）；The Hong Kong University of Science and Technology（香港科技大学）
Beyond the Surface: Measuring Self-Preference in LLM Judgments
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学首都人工智能学院）；Huawei Poisson Lab（华为Poisson实验室）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理重点实验室）
Exploring LLM-based Data Annotation Strategies for Medical Dialogue Preference Alignment
School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies(PKU), MOE, China（教育部高可信软件技术重点实验室（北京大学））；Beijing Jiaotong University（北京交通大学）
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
OpenGVLab（开放智能机器研究实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）
Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Princeton University（普林斯顿大学）；Nanyang Technological University（南洋理工大学）；Salesforce Research（Salesforce 研究）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore（高性能计算研究所，新加坡科技研究局）
Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task
Stanford University（斯坦福大学）；Australian National University（澳大利亚国立大学）；University of St. Andrews（圣安德鲁斯大学）；University of Cambridge（剑桥大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Department of Computer Science, University of Illinois at Chicago（芝加哥大学计算机科学系）
Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists
Paul G. Allen School of Computer Science \& Engineering University of Washington Seattle WA USA；The Information School University of Washington Allen Institute for AI Seattle WA USA；Allen Institute for AI Seattle WA USA；University of Washington；Allen Institute for AI
