潜在空间中的评判：通过语义保持压缩实现高效生成式奖励建模

中文摘要 AI 辅助

奖励建模通常需要联合表示和推理多个评估标准，然而逐词地语言化这一过程可能会产生大量的推理成本。最近关于潜在推理的研究表明，连续状态可能更紧凑地支持这种计算。我们引入了LatentGRM，一个基于语义分块、压缩和重建的潜在评估框架。通过利用基于评分标准的评估结构来指导压缩，LatentGRM学习了紧凑的连续轨迹，这些轨迹支持无需生成文本评估的自主成对判断。一个独立的解释器从这些轨迹中重建评估文本，提供了压缩下保留信息的离线视图。在匹配的训练数据和骨干网络下，LatentGRM在4B和8B规模上相对于显式监督微调（SFT）评判器取得了具有竞争力的聚合偏好准确性。在四个基准域中，LatentGRM-8B在vote@5时将评估轨迹压缩了8.9--9.2倍，并将总评判推理时间减少了6.1--7.0倍。受控的评分标准干预表明，依赖于标准的选择偏好信息通过潜在序列传递。这些结果共同表明，连续潜在评估可以在保持竞争性判断质量的同时显著降低推理成本。

英文摘要

Reward modeling often requires jointly representing and reasoning over multiple evaluation criteria, yet verbalizing this process token by token can incur substantial inference cost. Recent work on latent reasoning suggests that continuous states may support this computation more compactly. We introduce LatentGRM, a latent evaluation framework built on semantic chunking, compression, and reconstruction. By using the structure of rubric-guided evaluations to guide compression, LatentGRM learns compact continuous trajectories that support autonomous pairwise judgments without generating textual assessments. A separate interpreter reconstructs evaluation text from these trajectories, providing an offline view of the information retained under compression. Under matched training data and backbones, LatentGRM achieves competitive aggregate preference accuracy relative to explicit Supervised Fine-Tuning (SFT) judges at both 4B and 8B scales. Across four benchmark domains, LatentGRM-8B compresses evaluation trajectories by 8.9--9.2x and reduces total judge inference time by 6.1--7.0x at vote@5. Controlled rubric interventions show that criterion-dependent preference information is carried through the latent sequence. Together, these results demonstrate that continuous latent evaluation can substantially reduce inference cost while preserving competitive judgment quality.

发表机构

• Soochow University（苏州大学）
• University of Cambridge（剑桥大学）
• Chalmers University of Technology（查尔姆斯理工大学）
• Peking University（北京大学）
• Tsinghua University（清华大学）
• The Hong Kong University of Science and Technology（香港科技大学）

机构由 AI 辅助整理，请以论文原文为准。