Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual representations, and align them with each other. This disentangled task-invariant alignment of representations facilitates grounding and knowledge transfer across both tasks. We show that the proposed model outperforms a range of baselines on both tasks in simulated 3D environments. We also show that this disentanglement of representations makes our model modular, interpretable, and allows for transfer to instructions containing new words by leveraging object detectors.
While few-shot learning as a transfer learning paradigm has gained significant traction for scenarios with limited data, it has primarily been explored in the context of building unimodal and unilingual models. Furthermore, a significant part of the existing literature in the domain of few-shot multitask learning perform in-context learning which requires manually generated prompts as the input, yielding varying outcomes depending on the level of manual prompt-engineering. In addition, in-context learning suffers from substantial computational, memory, and storage costs which eventually leads …
Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework
ShanghaiTech University（上海科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Shanghai for Science and Technology（上海理工大学）；The University of Manchester（曼彻斯特大学）；University College London（伦敦大学学院）
GenRL: Multimodal-foundation world models for generalization in embodied agents
IDLab, Ghent University（IDLab，根特大学）；VERSES AI Research Lab（VERSES AI研究实验室）；Mila, University of Montreal（米拉研究所，蒙特利尔大学）；ServiceNow Research（ServiceNow研究院）
MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
School of Computer Science, Peking University（北京大学计算机科学学院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Large Multimodal Models for Embodied Intelligent Driving: The Next Frontier in Self-Driving?
School of Information and Electrical Engineering, Hebei University of Engineering（河北工程大学信息与电子工程学院）；School of Information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Department of Electrical and Computer Engineering, Auburn University（阿肯色大学电气与计算机工程系）；Department of Electrical and Computer Engineering, University of Houston（休斯顿大学电气与计算机工程系）；Department of Computer Science and Engineering, Kyung Hee University（庆熙大学计算机科学与工程系）；Machine Learning Department, Mohamed Bin Zayed University of Artificial Intelligence（Mohamed Bin Zayed人工智能大学机器学习系）
Learning a good representation is an essential component for deep reinforcement learning (RL). Representation learning is especially important in multitask and partially observable settings where building a representation of the unknown environment is crucial to solve the tasks. Here we introduce Prediction of Bootstrap Latents (PBL), a simple and flexible self-supervised representation learning algorithm for multitask deep RL. PBL builds on multistep predictive representations of future observations, and focuses on capturing structured information about environment dynamics. Specifically, PBL…
University of California, Riverside（加州大学河滨分校）；Mitsubishi Electric Research Labs（三菱电机研究实验室）
Recent years have seen embodied visual navigation advance in two distinct directions: (i) in equipping the AI agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-visual navigation. However, the real world is not only multimodal, but also often complex, and thus in spite of these advances, agents still need to understand the uncertainty in their actions and seek instructions to navigate. To this end, we present AVLEN~ -- an interactive agent for Audio-Visual-Language Embodied Navigation. Similar to audio-visual navigation tasks, the goal…
Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction
Multimodal Art Projection Research Community（多模态艺术投影研究社区）；Beijing University of Posts and Telecommunications（北京邮电大学）；HKUST（香港科技大学）；University of Waterloo（滑铁卢大学）；Vector Institute（矢量研究所）
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Sensetime Group Limited（商汤集团有限公司）；Beihang University（北京航空航天大学）
Forms are a common type of document in real life and carry rich information through textual contents and the organizational structure. To realize automatic processing of forms, word grouping and relation extraction are two fundamental and crucial steps after preliminary processing of optical character reader (OCR). Word grouping is to aggregate words that belong to the same semantic entity, and relation extraction is to predict the links between semantic entities. Existing works treat them as two individual tasks, but these two tasks are correlated and can reinforce each other. The grouping pr…
Amazon Alexa AI（亚马逊Alexa人工智能）；University of Southern California（南加州大学）；University of California, San Diego（加州大学圣迭戈分校）；University of California, Los Angeles（加州大学洛杉矶分校）
Recent years have witnessed an emerging paradigm shift toward embodied artificial intelligence, in which an agent must learn to solve challenging tasks by interacting with its environment. There are several challenges in solving embodied multimodal tasks, including long-horizon planning, vision-and-language grounding, and efficient exploration. We focus on a critical bottleneck, namely the performance of planning and navigation. To tackle this challenge, we propose a Neural SLAM approach that, for the first time, utilizes several modalities for exploration, predicts an affordance-aware semanti…
Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. We use both types of pairs in MURAL (MUltimodal, MUltitask Representations Across Languages), a dual encoder that solves two tasks: 1) image-text matching and 2) translation pair matching. By incorporating billions of translation pairs, MURAL extends ALIGN (Jia et al. PMLR'21)--a state-of-the-art dual encoder learned from 1.8 billion noisy image-text pairs. When using the same encoders, MURAL's performance matches or exceeds ALIGN's cross-modal retrieval performa…
Given the massive market of advertising and the sharply increasing online multimedia content (such as videos), it is now fashionable to promote advertisements (ads) together with the multimedia content. It is exhausted to find relevant ads to match the provided content manually, and hence, some automatic advertising techniques are developed. Since ads are usually hard to understand only according to its visual appearance due to the contained visual metaphor, some other modalities, such as the contained texts, should be exploited for understanding. To further improve user experience, it is nece…
Robotics at Google（谷歌机器人部门）；TU Berlin（柏林工业大学）；Google Research（谷歌研究院）
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, f…
A Smart-Glasses for Emergency Medical Services via Multimodal Multitask Learning
Computer Engineering, Texas A\&M University 3 Texas A\&M University Emergency Medical Services (EMS), 4 Biomedical Data Science, Stanford University, 5 Texas A\&M School of Public Health
U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection
University of Cambridge（剑桥大学）；the Alan Turing Institute（艾伦·图灵研究所）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Middle East Technical University (METU)（中东技术大学）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Google Health（谷歌健康）
Metadata are general characteristics of the data in a well-curated and condensed format, and have been proven to be useful for decision making, knowledge discovery, and also heterogeneous data organization of biobank. Among all data types in the biobank, pathology is the key component of the biobank and also serves as the gold standard of diagnosis. To maximize the utility of biobank and allow the rapid progress of biomedical science, it is essential to organize the data with well-populated pathology metadata. However, manual annotation of such information is tedious and time-consuming. In the…
FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Prediction
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Medical Big Data Research Center, Chinese PLA General Hospital（中国人民解放军总医院医学大数据中心）
Deep multimodal saliency parcellation of cerebellar pathways: linking microstructure and individual function through explainable multitask learning
University of New South Wales（新南威尔士大学）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）；Harvard University（哈佛大学）；Massachusetts General Hospital, Harvard Medical School（哈佛医学院麻省总医院）；Boston University School of Medicine（波士顿大学医学院）；University of Electronic Science and Technology of China（电子科技大学）；Colby College（科尔比学院）
Flood-DamageSense: Multimodal Mamba with Multitask Learning for Building Flood Damage Assessment using SAR Remote Sensing Imagery
Urban Resilience.AI Lab（城市韧性人工智能实验室）；Zachry Department of Civil and Environmental Engineering（扎克里土木与环境工程系）；Texas A&M University（德克萨斯A&M大学）；College Station, TX（学院站, 德克萨斯）
Agent AI: Surveying the Horizons of Multimodal Interaction
Microsoft Gaming（微软游戏）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Washington（华盛顿大学）；Microsoft Research, Redmond（微软研究院（雷德蒙德））；Stanford University（斯坦福大学）
Motivated by recent findings from cognitive neural science, we advocate the use of a dual-level model for concept representations: the embodied level consists of concept-oriented feature representations, and the symbolic level consists of concept graphs. Embodied concept representations are modality specific and exist in the form of feature vectors in a feature space. Symbolic concept representations, on the other hand, are amodal and language specific, and exist in the form of word / knowledge-graph embeddings in a concept / knowledge space. The human conceptual system comprises both embodied…
