We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen. Prior works modeled multimodal UI grounding in one round: the user gives a command and the agent responds to the command. Yet, in a realistic scenario, a user command can be ambiguous when the target action is inherently difficult to articulate in natural language. MUG allows multiple rounds of interactions such that upon seeing the agent responses, the user can give further commands for the agent to refine or even correct its actions. Such interaction is critical for improving grounding performances in real-world use cases. To investigate the problem, we create a new dataset that consists of 77,820 sequences of human user-agent interaction on mobile interfaces in which 20% involves multiple rounds of interactions. To establish our benchmark, we experiment with a range of modeling variants and evaluation strategies, including both offline and online evaluation-the online strategy consists of both human evaluation and automatic with simulators. Our experiments show that allowing iterative interaction significantly improves the absolute task completion by 18% over the entire test dataset and 31% over the challenging subset. Our results lay the foundation for further investigation of the problem.
PAYADOR: A Minimalist Approach to Grounding Language Models on Structured Data for Interactive Storytelling and Role-playing Games
Instituto de Computación, Facultad de Ingeniería, Universidad de la República(乌拉圭共和国大学工程学院计算研究所);Facultad de Informática, Universidad Complutense de Madrid(马德里康普顿斯大学信息学院)
DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards
York University(约克大学);RBC;Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);Nanyang Technological University(南洋理工大学);Salesforce Research(Salesforce研究)
AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
Yonsei University College of Medicine(延世大学医学院);Yonsei Institute for Digital Health(延世大学数字健康研究所);Yonsei University(延世大学);Institute of Behavioral Sciences in Medicine(医学行为科学研究所)
Knowledge-enhanced Agents for Interactive Text Games
Information Sciences Institute, University of Southern California(南加州大学信息科学研究所);Bosch Center for Artificial Intelligence(博世人工智能中心);Carnegie Mellon University(卡内基梅隆大学)
SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Southern California(南加州大学);University of Washington(华盛顿大学);Tsinghua University(清华大学);Paradigm Inc.(第四范式公司);University of California, San Diego(加州大学圣地亚哥分校);MosaicML(MosaicML公司)