DigiData: Training and Evaluating General-Purpose Mobile Control Agents
FAIR at Meta(Meta 的 FAIR 研究组);Meta Reality Labs(Meta 现实实验室);University of Southern California(南加州大学)
Yuxuan Sun, Manchen Wang, Shengyi Qian, William R. Wong, Eric Gan, Pierluca D'Oro, Alejandro Castillejo Munoz, Sneha Silwal, Pedro Matias, Nitin Kamra, Satwik K… 展开作者
Yuxuan Sun, Manchen Wang, Shengyi Qian, William R. Wong, Eric Gan, Pierluca D'Oro, Alejandro Castillejo Munoz, Sneha Silwal, Pedro Matias, Nitin Kamra, Satwik Kottur, Nick Raines, Xuanyi Zhao, Joy Chen, Joseph Greer, Andrea Madotto, Allen Bolourchi, James Valori, Kevin Carlberg, Karl Ridgeway, Joseph Tighe
AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building blocks are essential: high-quality datasets that enable agents to achieve complex and human-relevant goals, and robust evaluation methods that allow researchers and practitioners to rapidly enhance agent performance. In this paper, we introduce DigiData, a large-scale, high-quality, diverse, multi-modal dataset designed for training mobile control agents. Unlike existing datasets, which derive goals from unstructured interactions, DigiData is meticulously constructed through comprehensive exploration of app features, resulting in greater diversity and higher goal complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment. Our contributions aim to significantly advance the development of mobile control agents, paving the way for more intuitive and effective human-device interactions.
Information Extraction and Human-Robot Dialogue towards Real-life Tasks: A Baseline Study with the MobileCS Dataset
Tsinghua University(清华大学);Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute(清华大学-中国移动通信集团有限公司联合研究院);China Mobile Research Institute(中国移动研究院)
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
XiaoMi AI Lab(小米AI实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of Electronic Science and Technology of China(电子科技大学)
Benchmarking LLM Judges for Mobile Agent Evaluation
Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Concordia University(康考迪亚大学);University of Toronto(多伦多大学);Shanghai University(上海大学);McMaster University(麦克马斯特大学)
Western Mediterranean wetlands bird species classification: evaluating small-footprint deep learning approaches on a new annotated dataset
La Salle-Universitat Ramon Llull(拉蒙·柳利大学拉萨尔学院)
部署一个运行在无线声学传感器网络上的专家系统,该网络由生物声学监测设备组成,可以从声音中识别鸟类,这将使许多具有生态价值的任务实现自动化,包括分析鸟类种群组成或检测环境利益领域的濒危物种。得益于人工智能的最新进展,赋予这些设备准确的音频分类功能是可能的,其中深度学习技术表现出色。然而,使生物声学设备负担得起的一个关键问题是使用可以嵌入在资源和电池受限的硬件平台中的小足迹深度神经网络。出于这个原因,这项工作提出了两个重型和大型足迹深度神经网络(VGG 16和ResNet 50)和轻量级替代方案MobileNetV2之间的关键比较分析。我们的实验结果表明,MobileNetV2的平均F1分数比ResNet50低不到5%(0.789 vs. 0.834),表现优于VGG16,占用空间小近40倍。此外,为了比较模型,我们创建并公开了西地中海湿地鸟类数据集,包括201.6分钟和5,795个Aiguamolls de l'Empordà自然公园20种特有鸟类的音频摘录。
Nanyang Technological University(南洋理工大学);MiLM Plus, Xiaomi Inc.(小米公司);Independent Researchers(独立研究人员);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Wuhan University(武汉大学);Xiamen University(厦门大学)
Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation
AIRI;Moscow Institute of Physics and Technology (MIPT)(莫斯科物理技术学院);Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);Sberbank, Robotics Center(Sberbank机器人中心);Skoltech
本文提出 Kitchen-R 基准,在 Isaac Sim 数字孪生厨房环境中统一评估任务规划与底层控制,并提供视觉语言模型规划、扩散策略控制及轨迹采集系统,支持独立与集成评估,弥补具身 AI 中语言引导机器人整体评测的空白。