Presentation and Analysis of a Multimodal Dataset for Grounded Language Learning
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；Booz Allen Hamilton（博思艾伦汉密尔顿）；Johns Hopkins Applied Physics Laboratory（约翰斯·霍普金斯应用物理实验室）
Patrick Jenkins, Rishabh Sachdeva, Gaoussou Youssouf Kebe, Padraig Higgins, Kasra Darvish, Edward Raff, Don Engel, John Winder, Francis Ferraro, Cynthia Matuszek
Grounded language acquisition -- learning how language-based interactions refer to the world around them -- is amajor area of research in robotics, NLP, and HCI. In practice the data used for learning consists almost entirely of textual descriptions, which tend to be cleaner, clearer, and more grammatical than actual human interactions. In this work, we present the Grounded Language Dataset (GoLD), a multimodal dataset of common household objects described by people using either spoken or written language. We analyze the differences and present an experiment showing how the different modalities affect language learning from human in-put. This will enable researchers studying the intersection of robotics, NLP, and HCI to better investigate how the multiple modalities of image, text, and speech interact, as well as show differences in the vernacular of these modalities impact results.
VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；National Key Laboratory of General Artificial Intelligence（通用人工智能全国重点实验室）
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；McMaster University（麦克马斯特大学）；Meta Platforms（Meta平台公司）
OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Institute of Technology（北京理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）
