LLM-as-a-Judge：迈向石板推荐系统的世界模型
Offline A/B Testing of Slate Recommendation Systems with LLMs: Reducing the Dependency on Pre-Collected User Interaction Data
AI总结：
本文研究大型语言模型如何通过对石板推荐结果的成对推理来建模用户偏好，并通过跨数据集的实证研究揭示了任务性能与LLM偏好函数特性之间的关系，展示了LLM作为推荐系统世界模型的潜力。
摘要：
跨领域建模用户偏好仍然是石板推荐（即推荐一个有序项目序列）研究中的一个关键挑战。我们研究大型语言模型（LLM）如何通过对石板的成对推理有效地充当用户偏好的世界模型。我们开展了一项实证研究，涉及多个LLM在跨越不同数据集的三个任务上的表现。我们的结果揭示了任务性能与LLM所捕捉的偏好函数特性之间的关系，这为改进方向提供了线索，并突显了LLM作为推荐系统中世界模型的潜力。
英文摘要：
Slate recommender systems (RecSys) present users with ordered sets of interacting items (e.g., playlists). We investigate whether large language models (LLMs) can articulate pairwise preferences between slates for synthetic A/B testing of slate RecSys. We introduce a validation protocol measuring the alignment of synthetic preferences with classical RecSys metrics and their compliance with preference axioms, and use it to characterise how LLM pre-training and configuration affect slate preference articulation. Combined with the generalized Rao-Kupper model, synthetic LLM-based A/B testing recovers rankings that remain stable across utility weightings, whereas off-policy estimators are reliable only when the target utility matches the logged behavior. We position it as a screening stage between off-policy evaluation and live experiments: not a replacement for A/B testing, but a way to reserve its cost for the most promising candidates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
