A Statistical Framework for Alignment with Biased AI Feedback
Center for Data Science, Zhejiang University（浙江大学数据科学中心）；Department of Electrical and Computer Engineering, Rutgers University（罗格斯大学电气与计算机工程系）；Department of Statistics, Rutgers University（罗格斯大学统计学系）；Faculty of Business and Economics, The University of Hong Kong（香港大学商学院）
Modern alignment pipelines are increasingly replacing expensive human preference labels with evaluations from large language models (LLM-as-Judge). However, AI labels can be systematically biased compared to high-quality human feedback datasets. In this paper, we develop two debiased alignment methods within a general framework that accommodates heterogeneous prompt-response distributions and external human feedback sources. Debiased Direct Preference Optimization (DDPO) augments standard DPO with a residual-based correction and density-ratio reweighting to mitigate systematic bias, while retaining DPO's computational efficiency. Debiased Identity Preference Optimization (DIPO) directly estimates human preference probabilities without imposing a parametric reward model. We provide theoretical guarantees for both methods: DDPO offers a practical and computationally efficient solution for large-scale alignment, whereas DIPO serves as a robust, statistically optimal alternative that attains the semiparametric efficiency bound. Empirical studies on sentiment generation, summarization, and single-turn dialogue demonstrate that the proposed methods substantially improve alignment efficiency and recover performance close to that of an oracle trained on fully human-labeled data.
On Mitigating Affinity Bias through Bandits with Evolving Biased Feedback
Algorithms and Randomness Center, Georgia Institute of Technology（佐治亚理工学院算法与随机性中心）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Research（谷歌研究院）
Scalable Valuation of Human Feedback through Provably Robust Model Alignment
The University of Osaka（大阪大学）；Lattice Lab, Toyota Motor Corporation（丰田公司Lattice实验室）；Machine Learning Research Group, University of Oxford（牛津大学机器学习研究组）；RIKEN AIP（理化学研究所AIP）
正在进行的研究已经确定了生物网络和深度人工神经网络中神经表征之间的相似性。这使得人们对开发用于训练人工网络的反向传播学习算法与大脑中的突触可塑性规则之间的类比重新产生了兴趣。这些努力受到反向传播的生物学上难以置信的特征的挑战，其中之一是对对称的前向和后向突触权重的依赖。已经提出了许多不依赖于权重对称的方法，但到目前为止，这些方法未能扩展到深度卷积网络和复杂数据。我们确定的主要障碍，这种算法的可扩展性，并介绍了几种技术来减轻它们。我们证明，修改后的反馈对齐方法，强制执行较弱形式的权重对称性，一个需要协议的重量符号，但不幅度，可以实现与反向传播的性能竞争。我们的研究结果补充了Bartunov et al.（2018）和Xiao et al.（2018 b）的研究结果，并表明促进前馈和反馈权重对齐的机制对于深度网络中的学习至关重要。
Langevin-Informed Transfer Learning: Replacing Target Samples by Black-Box Feedback
Istituto Italiano di Tecnologia（意大利理工学院）；University of Novi Sad（诺维萨德大学）；CMAP-Ecole Polytechnique（巴黎综合理工学院应用数学中心）；Télécom Sud-Paris（巴黎南电信学院）；Université Paris Nanterre（巴黎南泰尔大学）；University College London（伦敦大学学院）
Align, then memorise: the dynamics of learning with feedback alignment
Ecole Normale Supérieure（巴黎高等师范学院）；EPFL（洛桑联邦理工学院）；Facebook AI Research（脸书人工智能研究院）；LightOn；International School of Advanced Studies (SISSA)（国际高等研究学院（SISSA））
