Anticipating future actions is a highly challenging task due to the diversity and scale of potential future actions; yet, information from different modalities help narrow down plausible action choices. Each modality can provide diverse and often complementary context for the model to learn from. While previous multi-modal methods leverage information from modalities such as video and audio, we primarily explore how text descriptions of actions and objects can also lead to more accurate action anticipation by providing additional contextual cues, e.g., about the environment and its contents. We propose a Multi-modal Contrastive Anticipative Transformer (M-CAT), a video transformer architecture that jointly learns from multi-modal features and text descriptions of actions and objects. We train our model in two stages, where the model first learns to align video clips with descriptions of future actions, and is subsequently fine-tuned to predict future actions. Compared to existing methods, M-CAT has the advantage of learning additional context from two types of text inputs: rich descriptions of future actions during pre-training, and, text descriptions for detected objects and actions during modality feature fusion. Through extensive experimental evaluation, we demonstrate that our model outperforms previous methods on the EpicKitchens datasets, and show that using simple text descriptions of actions and objects aid in more effective action anticipation. In addition, we examine the impact of object and action information obtained via text, and perform extensive ablations.
Action anticipation in egocentric videos is a difficult task due to the inherently multi-modal nature of human actions. Additionally, some actions happen faster or slower than others depending on the actor or surrounding context which could vary each time and lead to different predictions. Based on this idea, we build upon RULSTM architecture, which is specifically designed for anticipating human actions, and propose a novel attention-based technique to evaluate, simultaneously, slow and fast features extracted from three different modalities, namely RGB, optical flow, and extracted objects. T…
University of Rochester（罗切斯特大学）；JD AI Research（京东人工智能研究院）
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation. Unlike previous anticipation tasks that aim at action label prediction, our work targets at generating natural language outputs that provide interpretable and accurate descriptions of future action steps. It is a challenging task due to the lack of semantic information extracted from the instructional videos. To overcome this challenge, we propose a novel knowledge distillation framework to exploit the related external textual knowledge …
Imperial College London（帝国理工学院）；Shanghai Jiao Tong University（上海交通大学）
In this report, the technical details of our submission to the EPIC-Kitchens Action Anticipation Challenge 2021 are given. We developed a hierarchical attention model for action anticipation, which leverages Transformer-based attention mechanism to aggregate features across temporal dimension, modalities, symbiotic branches respectively. In terms of Mean Top-5 Recall of action, our submission with team name ICL-SJTU achieved 13.39% for overall testing set, 10.05% for unseen subsets and 11.88% for tailed subsets. Additionally, it is noteworthy that our submission ranked 1st in terms of verb cla…
Fraunhofer IOSB（弗劳恩霍夫光学、电子与精密工程研究所）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
Although human action anticipation is a task which is inherently multi-modal, state-of-the-art methods on well known action anticipation datasets leverage this data by applying ensemble methods and averaging scores of unimodal anticipation networks. In this work we introduce transformer based modality fusion techniques, which unify multi-modal data at an early stage. Our Anticipative Feature Fusion Transformer (AFFT) proves to be superior to popular score fusion approaches and presents state-of-the-art results outperforming previous methods on EpicKitchens-100 and EGTEA Gaze+. Our model is eas…
QueryMamba: A Mamba-Based Encoder-Decoder Architecture with a Statistical Verb-Noun Interaction Module for Video Action Forecasting @ Ego4D Long-Term Action Anticipation Challenge 2024
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Fraunhofer IOSB（弗劳恩霍夫IOSB研究所）
Revolutionizing Disease Diagnosis with simultaneous functional PET/MR and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks
School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Zhongshan Hospital（中山医院）；State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University（上海科技大学先进医用材料与器件国家重点实验室）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
Egocentric action anticipation consists in understanding which objects the camera wearer will interact with in the near future and which actions they will perform. We tackle the problem proposing an architecture able to anticipate actions at multiple temporal scales using two LSTMs to 1) summarize the past, and 2) formulate predictions about the future. The input video is processed considering three complimentary modalities: appearance (RGB), motion (optical flow) and objects (object-based features). Modality-specific predictions are fused using a novel Modality ATTention (MATT) mechanism whic…
From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学LUT）；School of Computing and Information Technology, Great Bay University（大湾区大学计算与信息技术学院）
本文提出端到端架构 ARR，将动作预判分解为动作识别与序列推理，并通过 NAP 和无监督解码器预训练学习动作统计关系，在三个数据集上验证了有效性。
Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?
Universidad de Alicante（阿利坎特大学）；Foundation for Research and Technology-Hellas（希腊基础研究与技术基金会）；University of Crete（克里特大学）；Hellenic Mediterranean University（希腊地中海大学）
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Southeast University - Monash University Joint Graduate School（东南大学-莫纳什大学联合研究生院）；School of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）
Recognizing sounds is a key aspect of computational audio scene analysis and machine perception. In this paper, we advocate that sound recognition is inherently a multi-modal audiovisual task in that it is easier to differentiate sounds using both the audio and visual modalities as opposed to one or the other. We present an audiovisual fusion model that learns to recognize sounds from weakly labeled video recordings. The proposed fusion model utilizes an attention mechanism to dynamically combine the outputs of the individual audio and visual models. Experiments on the large scale sound events…
EVA: An Embodied World Model for Future Video Anticipation
Hong Kong University of Science（香港科学与技术大学）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）
本文提出 RoG 推理策略、EVA-Bench 基准及多阶段训练的 EVA 具身世界模型，以提升多步、OOD 和长序列未来视频预测能力。
Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving
Aalto University（阿尔托大学）；Finnish Center for Artificial Intelligence（芬兰人工智能中心）；Gwangju Institute of Science and Technology（光州科学技术院）；University of Alberta（阿尔伯塔大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；Polish Academy of Sciences（波兰科学院）
Many recent datasets contain a variety of different data modalities, for instance, image, question, and answer data in visual question answering (VQA). When training deep net classifiers on those multi-modal datasets, the modalities get exploited at different scales, i.e., some modalities can more easily contribute to the classification results than others. This is suboptimal because the classifier is inherently biased towards a subset of the modalities. To alleviate this shortcoming, we propose a novel regularization term based on the functional entropy. Intuitively, this term encourages to b…
Multimodal Large Models Are Effective Action Anticipators
College of Information and Control Engineering, Xi’an University of Architecture and Technology（西安建筑科技大学信息与控制工程学院）；School of Astronautics, Northwestern Polytechnical University（西北工业大学航天学院）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）
Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities
University of Science and Technology of China（中国科学技术大学）；Tianjin University（天津大学）；Beijing Tiantan Hospital, Capital Medical University（首都医科大学附属北京天坛医院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Lenovo Research（联想研究院）；Lenovo Ltd.（联想集团有限公司）
Liver cancer is one of the most common cancers worldwide. Due to inconspicuous texture changes of liver tumor, contrast-enhanced computed tomography (CT) imaging is effective for the diagnosis of liver cancer. In this paper, we focus on improving automated liver tumor segmentation by integrating multi-modal CT images. To this end, we propose a novel mutual learning (ML) strategy for effective and robust multi-modal liver tumor segmentation. Different from existing multi-modal methods that fuse information from different modalities by a single model, with ML, an ensemble of modality-specific mo…
State Grid Shaanxi Electric Power Company Limited（国网陕西省电力公司）；State Grid Corporation of China（国家电网有限公司）；University of Science and Technology of China（中国科学技术大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；Baidu Inc（百度公司）
University of Oxford（牛津大学）；Fuzhou University（福州大学）；Fudan University（复旦大学）
Multi-modality cardiac imaging plays a key role in the management of patients with cardiovascular diseases. It allows a combination of complementary anatomical, morphological and functional information, increases diagnosis accuracy, and improves the efficacy of cardiovascular interventions and clinical outcomes. Fully-automated processing and quantitative analysis of multi-modality cardiac images could have a direct impact on clinical research and evidence-based patient management. However, these require overcoming significant challenges including inter-modality misalignment and finding optima…
Imperial College London（伦敦帝国理工学院）；The Chinese University of Hong Kong（香港中文大学）
Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model,…
Honda Research Institute USA（美国本田研究所）；University of California, Berkeley（加州大学伯克利分校）；Sungkyunkwan University（成均馆大学）
Predicting future trajectories of traffic agents in highly interactive environments is an essential and challenging problem for the safe operation of autonomous driving systems. On the basis of the fact that self-driving vehicles are equipped with various types of sensors (e.g., LiDAR scanner, RGB camera, radar, etc.), we propose a Cross-Modal Embedding framework that aims to benefit from the use of multiple input modalities. At training time, our model learns to embed a set of complementary features in a shared latent space by jointly optimizing the objective functions across different types …
Honda Research Institute USA（美国本田研究所）；University of California, Berkeley（加州大学伯克利分校）；Sungkyunkwan University（成均馆大学）
Predicting future trajectories of traffic agents in highly interactive environments is an essential and challenging problem for the safe operation of autonomous driving systems. On the basis of the fact that self-driving vehicles are equipped with various types of sensors (e.g., LiDAR scanner, RGB camera, radar, etc.), we propose a Cross-Modal Embedding framework that aims to benefit from the use of multiple input modalities. At training time, our model learns to embed a set of complementary features in a shared latent space by jointly optimizing the objective functions across different types …
Anticipating human actions is an important task that needs to be addressed for the development of reliable intelligent agents, such as self-driving cars or robot assistants. While the ability to make future predictions with high accuracy is crucial for designing the anticipation approaches, the speed at which the inference is performed is not less important. Methods that are accurate but not sufficiently fast would introduce a high latency into the decision process. Thus, this will increase the reaction time of the underlying system. This poses a problem for domains such as autonomous driving,…
Renmin University of China（中国人民大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, …
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation
Nanyang Technological University（南洋理工大学）；South China University of Technology（华南理工大学）；University of Oxford（牛津大学）；The University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）
School of Computer Science, Southeast University（东南大学计算机科学与工程学院）
Most of existing RGB-D salient object detection (SOD) methods follow the CNN-based paradigm, which is unable to model long-range dependencies across space and modalities due to the natural locality of CNNs. Here we propose the Hierarchical Cross-modal Transformer (HCT), a new multi-modal transformer, to tackle this problem. Unlike previous multi-modal transformers that directly connecting all patches from two modalities, we explore the cross-modal complementarity hierarchically to respect the modality gap and spatial discrepancy in unaligned regions. Specifically, we propose to use intra-modal…
Benchmark Evaluation of Federated Learning on Multi-organ Images
Hunan Provincial Key Laboratory on Bioinformatics, School of Computer Science and Engineering, Central South University（湖南生物信息省重点实验室，计算机科学与工程学院，中南大学）；Xinjiang Engineering Research Center of Big Data and Intelligent Software, School of software, Xinjiang University（新疆大数据与智能软件工程研究中心，软件学院，新疆大学）
University of Surrey（萨里大学）；iFlyTek-Surrey Joint Research Centre on Artificial Intelligence（讯飞-萨里人工智能联合研究中心）；University of Edinburgh（爱丁堡大学）
Self-supervised learning has gained prominence due to its efficacy at learning powerful representations from unlabelled data that achieve excellent performance on many challenging downstream tasks. However supervision-free pre-text tasks are challenging to design and usually modality specific. Although there is a rich literature of self-supervised methods for either spatial (such as images) or temporal data (sound or text) modalities, a common pre-text task that benefits both modalities is largely missing. In this paper, we are interested in defining a self-supervised pre-text task for sketche…
In this paper, we re-examine the task of cross-modal clip-sentence retrieval, where the clip is part of a longer untrimmed video. When the clip is short or visually ambiguous, knowledge of its local temporal context (i.e. surrounding video segments) can be used to improve the retrieval performance. We propose Context Transformer (ConTra); an encoder architecture that models the interaction between a video clip and its local temporal context in order to enhance its embedded representations. Importantly, we supervise the context transformer using contrastive losses in the cross-modal embedding s…
Renmin University of China（中国人民大学）；Beijing Visionary Intelligence Ltd.（北京维视智联科技有限公司）；Peking Union Medical College Hospital（北京协和医院）
This paper attacks an emerging challenge of multi-modal retinal disease recognition. Given a multi-modal case consisting of a color fundus photo (CFP) and an array of OCT B-scan images acquired during an eye examination, we aim to build a deep neural network that recognizes multiple vision-threatening diseases for the given case. As the diagnostic efficacy of CFP and OCT is disease-dependent, the network's ability of being both selective and interpretable is important. Moreover, as both data acquisition and manual labeling are extremely expensive in the medical domain, the network has to be re…
Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院人形与机器人研究所）；Mercedes-Benz Tech Innovation（梅赛德斯-奔驰技术创新公司）；School of Robotics, Hunan University（湖南大学机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）
UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving
Harbin Institute of Technology（哈尔滨工业大学）；Tianyijiaotong Technology Ltd.（天宜交通科技有限公司）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）
Recent studies suggest that combined analysis of Magnetic resonance imaging~(MRI) that measures brain atrophy and positron emission tomography~(PET) that quantifies hypo-metabolism provides improved accuracy in diagnosing Alzheimer's disease. However, such techniques are limited by the availability of corresponding scans of each modality. Current work focuses on a cross-modal approach to estimate FDG-PET scans for the given MR scans using a 3D U-Net architecture. The use of the complete MR image instead of a local patch based approach helps in capturing non-local and non-linear correlations be…
Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation
University of Copenhagen（哥本哈根大学）；ETH Zurich（苏黎世联邦理工学院）；Nankai University（南开大学）；University of Oxford（牛津大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）
MMLF: Multi-modal Multi-class Late Fusion for Object Detection with Uncertainty Estimation
Glasgow College, University of Electronic Science and Technology of China（电子科技大学格拉斯哥学院）；School of Automation Engineering, University of Electronic Science and Technology of China（电子科技大学自动化工程学院）
