通过严格估计分离缓解偏好泄漏以规范生成排序
Mitigating Preference Leakage via Strict Estimator Separation for Normative Generative Ranking
AI总结:
本文提出通过严格分离监督与评估模型,缓解偏好泄漏问题,提升生成排序的可信度。
摘要:
在生成信息检索(GenIR)中,瓶颈从生成转向了候选人的选择,特别是对于规范性标准如文化相关性。当前LLM作为裁判的评估方法常受到循环和偏好泄漏的影响,其中重叠的监督和评估模型会夸大性能。我们通过将文化相关性定义为查询内的排序任务,并引入一个无泄漏的双裁判框架,严格分离监督(裁判B)和评估(裁判A)。在新的基准测试集(33,052个文化相关的故事情报,NGR-33k)上,我们发现虽然经典基线只能获得有限的提升,但从受裁判B监督的交叉编码器中密集提取的BGE-M3模型在无泄漏的裁判A评估下表现显著优于交叉编码器。尽管交叉编码器为蒸馏提供了强大的监督信号,但蒸馏后的BGE-M3模型在无泄漏的裁判A评估下仍表现更优。我们在人工整理的道德故事数据集上验证了我们的框架,显示与人类规范有很强的一致性。我们的结果表明,严格分离评估者是可信GenIR评估的先决条件,证明了微妙的文化偏好可以被蒸馏成高效的排序器,而无需泄漏。
英文摘要:
In Generative Information Retrieval (GenIR), the bottleneck has shifted from generation to the selection of candidates, particularly for normative criteria such as cultural relevance. Current LLM-as-a-Judge evaluations often suffer from circularity and preference leakage, where overlapping supervision and evaluation models inflate performance. We address this by formalising cultural relevance as a within-query ranking task and introducing a leakage-free two-judge framework that strictly separates supervision (Judge B) from evaluation (Judge A). On a new benchmark of 33,052 (NGR-33k) culturally grounded stories, we find that while classical baselines yield only modest gains, a dense bi-encoder distilled from a Judge-B-supervised Cross-Encoder is highly effective. Although the Cross-Encoder provides a strong supervision signal for distillation, the distilled BGE-M3 model substantially outperforms it under leakage-free Judge~A evaluation. We validate our framework on the human-curated Moral Stories dataset, showing strong alignment with human norms. Our results demonstrate that rigorous evaluator separation is a prerequisite for credible GenIR evaluation, proving that subtle cultural preferences can be distilled into efficient rankers without leakage.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。