超越原始参与度：Netflix 推荐系统的反事实可观测性框架

中文摘要 AI 辅助

理解大规模推荐系统的性能仍然是一个未被充分探索的挑战，尤其是对于内容创作者和模型开发者而言。他们可获得的原始参与度信号，如观看次数和点击次数，混淆了内容质量、模型行为、展示偏差和受众覆盖范围，使得难以将结果归因于正确的起因。在这项工作中，我们提出了一个通用评估框架，该框架增强了Netflix多个推荐系统的可观测性，并通过多次生产部署证明了其有效性。该框架将推荐系统的可观测性视为一个反事实测量问题：即估计在缺少特定内容项或模型决策的情况下，推荐系统本会做什么，以及随后本会产生什么样的参与度。我们阐述了面向内容创作者和模型开发者的三个以利益相关者为中心的可观测性原则，并提出了涵盖偏差减少、相对性和增量性的测量方法，这些方法适用于单阶段和级联推荐系统，并从一个统一的测量基础服务于这两类受众。

英文摘要

Understanding the performance of large-scale recommender systems remains an underexplored challenge, especially for content creators and model developers. The raw engagement signals available to them, such as views and clicks, conflate content quality, model behavior, presentation bias, and audience reach, making it hard to attribute outcomes to the right cause. In this work, we present a general evaluation framework that enhances observability across multiple recommender systems at Netflix and demonstrate its effectiveness through several production deployments. The framework treats recommender-system observability as a counterfactual measurement problem: estimating what the recommender would have done, and what engagement would have followed, in the absence of a specific content item or model decision. We articulate three stakeholder-centered observability principles for content creators and model developers, and propose measurement methodologies covering bias reduction, relativity, and incrementality, applicable to both single-stage and cascading recommender systems and serving both audiences from a single measurement foundation.

发表机构

• Netflix（奈飞）

机构由 AI 辅助整理，请以论文原文为准。