大型推理模型是好的翻译评估器吗？分析与性能提升
Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
AI总结：
首次系统分析大型推理模型作为机器翻译评估器的潜力，发现其需定制评估材料、易过度思考且评分机制有高估问题，提出通过合成人类式思考轨迹校准模型，在WMT24基准上大幅减少思考预算并提升评估性能。
摘要：
大型推理模型（LRMs）的最新进展在生成最终答案之前引入了中间“思考”过程，提高了它们在复杂下游任务上的推理能力。然而，LRMs作为机器翻译（MT）质量评估器的潜力仍未得到充分探索。我们首次对LRM-as-a-judge在MT评估中进行了系统分析。我们识别出关键挑战，揭示LRMs需要量身定制的评估材料，倾向于对简单实例“过度思考”，并且存在导致高估的评分机制问题。为解决这些问题，我们提出通过训练LRMs于合成的人类式思考轨迹来校准其思考过程。我们在WMT24 Metrics基准上的实验表明，该方法大幅减少了约35倍的思考预算，同时在不同LRM规模（从7B到32B）上提升了评估性能（例如，R1-Distill-Qwen-7B实现了+8.7的相关性点提升）。这些发现凸显了高效校准的LRMs在推进细粒度自动MT评估方面的潜力。
英文摘要：
Recent advancements in large reasoning models (LRMs) have introduced an intermediate "thinking" process prior to generating final answers, improving their reasoning capabilities on complex downstream tasks. However, the potential of LRMs as evaluators for machine translation (MT) quality remains underexplored. We provides the first systematic analysis of LRM-as-a-judge in MT evaluation. We identify key challenges, revealing LRMs require tailored evaluation materials, tend to "overthink" simpler instances and have issues with scoring mechanisms leading to overestimation. To address these, we propose to calibrate LRM thinking by training them on synthetic, human-like thinking trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this approach largely reduces thinking budgets by ~35x while concurrently improving evaluation performance across different LRM scales from 7B to 32B (e.g., R1-Distill-Qwen-7B achieves a +8.7 correlation point improvement). These findings highlight the potential of efficiently calibrated LRMs to advance fine-grained automatic MT evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
