Crowd-SFT：面向大语言模型对齐的众包方法
Crowd-SFT: Crowdsourcing for LLM Alignment
AI总结：
本文提出开放众包微调框架Crowd-SFT，通过基于Shapley值的积分奖励机制和迭代多模型选择，在降低对齐偏差与成本的同时实现公平可扩展的LLM对齐。
摘要：
大语言模型（LLMs）越来越依赖监督微调（SFT）和基于人类反馈的强化学习（RLHF）来使模型响应与人类偏好对齐。RLHF采用强化学习方法并配合独立的奖励模型，而SFT则使用人工整理的数据集进行监督学习。这两种方法传统上都依赖规模较小且经过审核的标注者群体，因而成本高昂、容易产生偏差，且可扩展性受限。我们提出一个开放的众包微调框架，通过支持在不进行大规模标注者培训的情况下为SFT收集更广泛的反馈来解决这些局限。该框架通过一个与Shapley值相关联的积分奖励机制促进激励公平，并通过迭代模型更新引导模型收敛。我们的多模型选择框架相比单模型选择在目标距离上最多实现55%的降低，使后续实验得以验证我们的积分奖励机制与Shapley值（一种公认的用于归因个体贡献的方法）高度一致，从而支持公平且可扩展的参与。
英文摘要：
Large Language Models (LLMs) increasingly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to align model responses with human preferences. While RLHF employs a reinforcement learning approach with a separate reward model, SFT uses human-curated datasets for supervised learning. Both approaches traditionally depend on small, vetted groups of annotators, making them costly, prone to bias, and limited in scalability. We propose an open, crowd-sourced fine-tuning framework that addresses these limitations by enabling broader feedback collection for SFT without extensive annotator training. Our framework promotes incentive fairness via a point-based reward system correlated with Shapley values and guides model convergence through iterative model updates. Our multi-model selection framework demonstrates up to a 55% reduction in target distance over single-model selection, enabling subsequent experiments that validate our point-based reward mechanism's close alignment with Shapley values (a well-established method for attributing individual contributions) thereby supporting fair and scalable participation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
