Children acquire language subconsciously by observing the surrounding world and listening to descriptions. They can discover the meaning of words even without explicit language knowledge, and generalize to novel compositions effortlessly. In this paper, we bring this ability to AI, by studying the task of Visually grounded Language Acquisition (VLA). We propose a multimodal transformer model augmented with a novel mechanism for analogical reasoning, which approximates novel compositions by learning semantic mapping and reasoning operations from previously seen compositions. Our proposed method, Analogical Reasoning Transformer Networks (ARTNet), is trained on raw multimedia data (video frames and transcripts), and after observing a set of compositions such as "washing apple" or "cutting carrot", it can generalize and recognize new compositions in new video frames, such as "washing carrot" or "cutting apple". To this end, ARTNet refers to relevant instances in the training data and uses their visual features and captions to establish analogies with the query image. Then it chooses the suitable verb and noun to create a new composition that describes the new image best. Extensive experiments on an instructional video dataset demonstrate that the proposed method achieves significantly better generalization capability and recognition accuracy compared to state-of-the-art transformer models.
Towards Human-Like Machine Comprehension: Few-Shot Relational Learning in Visually-Rich Documents
School of Computer Engineering and Science, Shanghai University（上海大学计算机工程与科学学院）；Graduate School of Informatics, Kyoto University（京都大学信息学研究科）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
Respond Beyond Language: A Benchmark for Video Generation in Response to Realistic User Intents
School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；Serendipity One Inc.（Serendipity One公司）
Inferring the Unspoken: Aligning Embodied Agents with Implicit Preferences
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Computer Science & Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China（北京理工大学扬子江地区研究院）；National Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）
