缺失的是：大型语言模型输出的可解释评分
What Is Missing: Interpretable Ratings for Large Language Model Outputs
AI总结：
本文提出WIM评分系统，通过自然语言反馈生成可解释的评分，提升大型语言模型输出的可解释性和学习信号质量。
摘要：
当前的大型语言模型（LLM）偏好学习方法，如近端策略优化和直接偏好优化，通过直接排名或数值评分来学习模型输出，这些排名具有主观性，单个由裁判直接选择的数值评分是自然语言质量的糟糕代理。我们引入了
英文摘要：
Current Large Language Model (LLM) preference learning methods such as Proximal Policy Optimization and Direct Preference Optimization learn from direct rankings or numerical ratings of model outputs, these rankings are subjective, and a single numerical rating chosen directly by a judge is a poor proxy for the quality of natural language, we introduce the What Is Missing (WIM) rating system to produce rankings from natural-language feedback, WIM integrates into existing training pipelines, can be combined with other rating techniques, and can be used as input to any preference learning method without changing the learning algorithm, to compute a WIM rating, a human or LLM judge writes feedback describing what the model output is missing, we embed the output and the feedback with a sentence embedding model and compute the cosine similarity between the resulting vectors, we empirically observe that, compared to discrete numerical ratings, WIM yields fewer ties and larger rating deltas, which improves the availability of a learning signal in pairwise preference data, we use interpretable in the following limited sense: for each scalar rating, we can inspect the judge's missing-information text that produced it, enabling qualitative debugging of the preference labels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
