使用具有配置文件感知能力的 LLM-as-a-Judge 评估播客推荐
Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge
AI总结：
本文提出了一种基于LLM作为离线评估器的两阶段配置文件感知框架，通过构建用户收听历史的自然语言配置文件来评估播客推荐质量，有效降低输入复杂性并提升可解释性，在对照研究中高保真匹配人类判断。
摘要：
评估个性化推荐仍然是一个核心挑战，特别是在播客等长音频领域，传统的离线指标存在曝光偏差，而A/B测试等在线方法成本高昂且在操作上受限。在本文中，我们提出了一个新颖的框架，利用大型语言模型（LLMs）作为离线评估器，以可扩展和可解释的方式评估播客推荐的质量。我们的两阶段配置文件感知方法首先构建从90天收听历史中提炼出的自然语言用户配置文件。这些配置文件总结了主题兴趣和行为模式，作为用户偏好的紧凑、可解释的表示。我们不是用原始数据提示LLM，而是使用这些配置文件提供高级的、语义丰富的上下文，使LLM能够更有效地推理用户兴趣与推荐剧集之间的一致性。这降低了输入复杂性并提高了可解释性。然后，基于配置文件与剧集的匹配程度，提示LLM提供细粒度的逐点和成对判断。在一项有47名参与者参与的对照研究中，我们的配置文件感知评估器以高保真度匹配了人类判断，并且优于或匹配了使用原始收听历史的变体。该框架为推荐系统中的迭代测试和模型选择实现了高效的、配置文件感知的评估。
英文摘要：
Evaluating personalized recommendations remains a central challenge, especially in long-form audio domains like podcasts, where traditional offline metrics suffer from exposure bias and online methods such as A/B testing are costly and operationally constrained. In this paper, we propose a novel framework that leverages Large Language Models (LLMs) as offline judges to assess the quality of podcast recommendations in a scalable and interpretable manner. Our two-stage profile-aware approach first constructs natural-language user profiles distilled from 90 days of listening history. These profiles summarize both topical interests and behavioral patterns, serving as compact, interpretable representations of user preferences. Rather than prompting the LLM with raw data, we use these profiles to provide high-level, semantically rich context-enabling the LLM to reason more effectively about alignment between a user's interests and recommended episodes. This reduces input complexity and improves interpretability. The LLM is then prompted to deliver fine-grained pointwise and pairwise judgments based on the profile-episode match. In a controlled study with 47 participants, our profile-aware judge matched human judgments with high fidelity and outperformed or matched a variant using raw listening histories. The framework enables efficient, profile-aware evaluation for iterative testing and model selection in recommender systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
