在无需人工标注偏好数据的情况下让大语言模型（LLM）与人类期望对齐是一个重要问题。本文提出一种利用对比提示对下响应对的输出概率来评估响应偏好的方法，该方法在LLaMA2-7B和LLaMA2-13B上相比RLAIF能取得更优性能。基于此，我们提出自动对齐方法Direct Large Model Alignment（DLMA，大模型直接对齐）：首先用对比提示对自动生成偏好数据，接着继续用对比提示对评估生成的偏好数据并计算自奖励分数，最后结合该自奖励分数使用DPO算法有效对齐LLM。实验阶段，我们的DLMA方法无需依赖人工标注的偏好数据即可超越RLHF方法。
英文摘要：
Aligning large language models (LLMs) with human expectations without human-annotated preference data is an important problem. In this paper, we propose a method to evaluate the response preference by using the output probabilities of response pairs under contrastive prompt pairs, which could achieve better performance on LLaMA2-7B and LLaMA2-13B compared to RLAIF. Based on this, we propose an automatic alignment method, Direct Large Model Alignment (DLMA). First, we use contrastive prompt pairs to automatically generate preference data. Then, we continue to evaluate the generated preference data using contrastive prompt pairs and calculate a self-rewarding score. Finally, we use the DPO algorithm to effectively align LLMs by combining this self-rewarding score. In the experimental stage, our DLMA method could surpass the \texttt{RLHF} method without relying on human-annotated preference data.
Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳研究院）；Institute of Artificial Intelligence (TeleAI), China Telecom Corp Ltd（中国电信人工智能研究院）；College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China（浙江大学集成电路学院）；Zhongguancun Academy, Beijing, China（中关村学院）
Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models
Northwestern University（西北大学）；Syracuse University（雪城大学）；Trine University（特里恩大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Carnegie Mellon University（卡内基梅隆大学）
Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillation
Lab of Artificial Intelligence for Education, East China Normal University, Shanghai 200062, China（教育人工智能实验室，华东师范大学，上海200062，中国）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University, Shanghai 200062, China（教育人工智能研究所，华东师范大学，上海200062，中国）；School of Computer Science and Technology, East China Normal University, Shanghai 200062, China（计算机科学与技术学院，华东师范大学，上海200062，中国）
HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models
Zhejiang University（浙江大学）；Byte Dance（字节跳动）；National University of Singapore（新加坡国立大学）；Angelalign Inc China（Angelalign中国公司）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）
Distillation versus Contrastive Learning: How to Train Your Rerankers
Kahlert School of Computing, University of Utah（犹他大学计算机学院）；Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）；Capital One, Inc（Capital One公司）；The University of Queensland（昆士兰大学）
Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding
National Taiwan University（国立台湾大学）
口语语言理解（Spoken language understanding，SLU）是机器理解人类语音以实现更好交互的重要任务。然而，来自自动语音识别器（ASR）的错误通常会损害理解性能。实际上，ASR系统可能不容易针对目标场景进行调整。因此，本文采用对比目标学习对ASR错误具有鲁棒性的话语表征，并在模型微调中结合监督对比学习和自升华进一步增强泛化能力。在三个基准数据集上的实验证明了该方法的有效性。
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
