arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.04798cs.HCcs.IR
收藏

Orbit:一个用于设计和评估多目标排序器的框架

Orbit: A Framework for Designing and Evaluating Multi-objective Rankers

Chenyang Yang, Tesi Xiao, Michael Shavlovsky, Christian Kästner, Tongshuang Wu

AI总结:

本文提出Orbit框架及交互系统,将目标置于多目标排序器设计过程中心,支持实时探索设计权衡,并通过用户研究验证其能提升决策质量和权衡意识。

摘要:

生产环境中的机器学习需要平衡多个目标:这在排序或推荐模型中尤为明显,其中用户参与度、满意度、多样性和新颖性等相互冲突的目标必须同时被考虑。然而,设计多目标排序器本质上是一个动态的棘手问题——不存在单一最优解,且需求会随时间演变。有效的设计需要跨职能团队之间的协作,以及对广泛信息的仔细分析。在这项工作中,我们引入了Orbit,一个面向目标中心的排序器构建与迭代(Objective-centric Ranker Building and Iteration)的概念框架。该框架将目标置于设计过程的中心,使其作为沟通的边界对象,并指导从业者进行设计和评估。我们将Orbit实现为一个交互式系统,使利益相关者能够直接与目标空间交互,并支持对设计权衡的实时探索和评估。我们通过一项涉及十二位行业从业者的用户研究对Orbit进行了评估,结果表明它支持高效的设计空间探索,带来更明智的决策,并增强了对多目标固有权衡的认识。Orbit(1)为任何多目标机器学习模型开辟了以目标为中心的设计过程的新机遇,同时(2)为未来设计提供了启示,推动从业者超越狭隘的以指标为中心或以示例为中心的思维模式。

英文摘要:

Machine learning in production needs to balance multiple objectives: This is particularly evident in ranking or recommendation models, where conflicting objectives such as user engagement, satisfaction, diversity, and novelty must be considered at the same time. However, designing multi-objective rankers is inherently a dynamic wicked problem -- there is no single optimal solution, and the needs evolve over time. Effective design requires collaboration between cross-functional teams and careful analysis of a wide range of information. In this work, we introduce Orbit, a conceptual framework for Objective-centric Ranker Building and Iteration. The framework places objectives at the center of the design process, to serve as boundary objects for communication and guide practitioners for design and evaluation. We implement Orbit as an interactive system, which enables stakeholders to interact with objective spaces directly and supports real-time exploration and evaluation of design trade-offs. We evaluate Orbit through a user study involving twelve industry practitioners, showing that it supports efficient design space exploration, leads to more informed decision-making, and enhances awareness of the inherent trade-offs of multiple objectives. Orbit (1) opens up new opportunities of an objective-centric design process for any multi-objective ML models, as well as (2) sheds light on future designs that push practitioners to go beyond a narrow metric-centric or example-centric mindset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑