MENLO：从偏好到熟练度——在47种语言上评估和建模原生质量
MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages
AI总结：
MENLO通过偏好评估和建模提升多语言LLM的原生质量，结合强化学习和多任务学习方法改进模型表现。
摘要：
确保大型语言模型（LLM）在多种语言上的响应具有原生质量具有挑战性。为了解决这一问题，我们引入了MENLO框架，该框架基于观众设计启发机制，将原生响应质量的评估过程化。使用MENLO，我们创建了一个包含6,423个由人类标注的提示-响应偏好对的数据集，覆盖四个质量维度，在47种语言变体中具有高标注者间一致性。我们的评估发现，零样本LLM判断者显著受益于成对评估和我们的结构化标注评分表，但仍在我们的数据集上表现不如人类标注者。我们通过微调与强化学习、奖励塑造和多任务学习方法展示了显著的改进。此外，我们表明通过强化学习训练的判断者可以作为生成奖励模型，以提高LLM的多语言熟练度，尽管与人类判断仍存在差异。我们的发现表明了可扩展的多语言评估和偏好对齐的有希望的方向。我们发布了我们的数据集和评估框架以支持进一步的多语言LLM评估研究（https://huggingface.co/datasets/facebook/menlo）.
英文摘要：
Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a dataset of 6,423 human-annotated prompt-response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties. Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, yet they still underperform human annotators on our dataset. We demonstrate substantial improvements through fine-tuning with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models to enhance LLMs' multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scalable multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation (https://huggingface.co/datasets/facebook/menlo).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
