生产环境中的机器学习需要平衡多个目标:这在排序或推荐模型中尤为明显,其中用户参与度、满意度、多样性和新颖性等相互冲突的目标必须同时被考虑。然而,设计多目标排序器本质上是一个动态的棘手问题——不存在单一最优解,且需求会随时间演变。有效的设计需要跨职能团队之间的协作,以及对广泛信息的仔细分析。在这项工作中,我们引入了Orbit,一个面向目标中心的排序器构建与迭代(Objective-centric Ranker Building and Iteration)的概念框架。该框架将目标置于设计过程的中心,使其作为沟通的边界对象,并指导从业者进行设计和评估。我们将Orbit实现为一个交互式系统,使利益相关者能够直接与目标空间交互,并支持对设计权衡的实时探索和评估。我们通过一项涉及十二位行业从业者的用户研究对Orbit进行了评估,结果表明它支持高效的设计空间探索,带来更明智的决策,并增强了对多目标固有权衡的认识。Orbit(1)为任何多目标机器学习模型开辟了以目标为中心的设计过程的新机遇,同时(2)为未来设计提供了启示,推动从业者超越狭隘的以指标为中心或以示例为中心的思维模式。
英文摘要:
Machine learning in production needs to balance multiple objectives: This is particularly evident in ranking or recommendation models, where conflicting objectives such as user engagement, satisfaction, diversity, and novelty must be considered at the same time. However, designing multi-objective rankers is inherently a dynamic wicked problem -- there is no single optimal solution, and the needs evolve over time. Effective design requires collaboration between cross-functional teams and careful analysis of a wide range of information. In this work, we introduce Orbit, a conceptual framework for Objective-centric Ranker Building and Iteration. The framework places objectives at the center of the design process, to serve as boundary objects for communication and guide practitioners for design and evaluation. We implement Orbit as an interactive system, which enables stakeholders to interact with objective spaces directly and supports real-time exploration and evaluation of design trade-offs. We evaluate Orbit through a user study involving twelve industry practitioners, showing that it supports efficient design space exploration, leads to more informed decision-making, and enhances awareness of the inherent trade-offs of multiple objectives. Orbit (1) opens up new opportunities of an objective-centric design process for any multi-objective ML models, as well as (2) sheds light on future designs that push practitioners to go beyond a narrow metric-centric or example-centric mindset.
Designing and Evaluating Next-Generation Learning Interfaces: Linking AI, HCI, and the Learning Sciences
Texas A&M University(德克萨斯农工大学);Virginia Tech(弗吉尼亚理工大学);North Carolina State University(北卡罗来纳州立大学);Utah State University(犹他州立大学);University of Auckland(奥克兰大学);University of Oxford(牛津大学);Carnegie Mellon University(卡内基梅隆大学)
"Shall We Dig Deeper?": Designing and Evaluating Strategies for LLM Agents to Advance Knowledge Co-Construction in Asynchronous Online Discussions
Hong Kong University of Science and Technology(香港科技大学);North Carolina State University(北卡罗来纳州立大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
Designing and Evaluating Granular Consent for Data Sharing in Cardiac Disease Prevention
Ludwig Boltzmann Institute for Digital Health and Prevention(路德维希·玻尔兹曼数字健康与预防研究所);LMU Munich(慕尼黑大学);SOCIUM Research Center on Inequality and Social Policy, University of Bremen(不来梅大学不平等与社会政策研究中心);Leibniz ScienceCampus Digital Public Health, University of Bremen(不来梅大学莱布尼茨数字公共卫生科学园区)
CaseMaster: Designing and Evaluating a Probe for Oral Case Presentation Training with LLM Assistance
School of Information Science and Technology ShanghaiTech University(上海科技大学信息科学与技术学院);Department of Clinical Medicine Shanghai Clinical Research and Trial Center(上海临床研究试验中心临床医学部)
Human-AI Interaction Alignment: Designing, Evaluating, and Evolving Value-Centered AI For Reciprocal Human-AI Futures
NYU Shanghai, New York University(纽约大学上海校区);Google(谷歌);Massachusetts Institute of Technologyy(麻省理工学院);Google, Paradigms of Intelligence(谷歌、智能范式);Carnegie Mellon University, New York University(卡内基梅隆大学、纽约大学);Carnegie Mellon University(卡内基梅隆大学);OpenAI;University of Washington(华盛顿大学);Google DeepMind(谷歌DeepMind)
Designing for Dignity while Driving: Interaction Needs of Blind and Low-Vision Passengers in Fully Automated Vehicles
Queensland University of Technology(昆士兰科技大学);School of Design, Queensland University of Technology(昆士兰科技大学设计学院);School of Computer Science, Queensland University of Technology(昆士兰科技大学计算机科学学院);Centre for Accident Research and Road Safety - Queensland (CARRS-Q), Queensland University of Technology(昆士兰交通事故与道路安全研究中心);Center for Future Mobility, Queensland University of Technology(昆士兰大学未来出行中心)
Who Thinks First? Designing Productive Friction with Engage-to-Unlock GenAI
ETH Zurich(苏黎世联邦理工学院);Google DeepMind(谷歌DeepMind);King’s College London(伦敦国王学院);Google(谷歌);Google Research(谷歌研究);University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学)
Designing an Evaluation Framework for Large Language Models in Astronomy Research
Space Telescope Science Institute(空间望远镜科学研究所);Johns Hopkins University(约翰斯·霍普金斯大学);Oregon State University(俄勒冈州立大学);Loyola University Maryland(马里兰州洛约拉大学);Stanford University(斯坦福大学);European Space Astronomy Centre(欧洲空间天文中心);Australian National University(澳大利亚国立大学);Columbia University(哥伦比亚大学)
PADTHAI-MM: Principles-based Approach for Designing Trustworthy, Human-centered AI using MAST Methodology
Arizona State University(亚利桑那州立大学);Aptima, Inc.(Aptima公司);U.S. Department of Homeland Security(美国国土安全部);Air Force Office of Scientific Research(空军科学研究办公室)