IEEE International Conference on Computer Vision (ICCV) Workshop on Large Scale Holistic Video Understanding. The datasets and code are available at https://github.com/vivoutlaw/tcbp
True understanding of videos comes from a joint analysis of all its modalities: the video frames, the audio track, and any accompanying text such as closed captions. We present a way to learn a compact multimodal feature representation that encodes all these modalities. Our model parameters are learned through a proxy task of inferring the temporal ordering of a set of unordered videos in a timeline. To this end, we create a new multimodal dataset for temporal ordering that consists of approximately 30K scenes (2-6 clips per scene) based on the "Large Scale Movie Description Challenge". We analyze and evaluate the individual and joint modalities on three challenging tasks: (i) inferring the temporal ordering of a set of videos; and (ii) action recognition. We demonstrate empirically that multimodal representations are indeed complementary, and can play a key role in improving the performance of many applications.
We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed EfficientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as \wifi fingerprints that range up to 120 action classes. …
Super Encoding Network: Recursive Association of Multi-Modal Encoders for Video Understanding
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；the School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Shanghai AI Laboratory（上海人工智能实验室）
Convolutional Neural Networks (CNNs) are successful deep learning models in the field of computer vision. To get the maximum advantage of CNN model for Human Action Recognition (HAR) using inertial sensor data, in this paper, we use 4 types of spatial domain methods for transforming inertial sensor data to activity images, which are then utilized in a novel fusion framework. These four types of activity images are Signal Images (SI), Gramian Angular Field (GAF) Images, Markov Transition Field (MTF) Images and Recurrence Plot (RP) Images. Furthermore, for creating a multimodal fusion framework …
The problem of multimodal intent and trajectory prediction for human-driven vehicles in parking lots is addressed in this paper. Using models designed with CNN and Transformer networks, we extract temporal-spatial and contextual information from trajectory history and local bird's eye view (BEV) semantic images, and generate predictions about intent distribution and future trajectory sequences. Our methods outperform existing models in accuracy, while allowing an arbitrary number of modes, encoding complex multi-agent scenarios, and adapting to different parking maps. To train and evaluate our…
We present MMFT-BERT(MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an…
Most approaches for semantic segmentation use only information from color cameras to parse the scenes, yet recent advancements show that using depth data allows to further improve performances. In this work, we focus on transformer-based deep learning architectures, that have achieved state-of-the-art performances on the segmentation task, and we propose to employ depth information by embedding it in the positional encoding. Effectively, we extend the network to multimodal data without adding any parameters and in a natural way that makes use of the strength of transformers' self-attention mod…
Probing Multimodal Fusion in the Brain: The Dominance of Audiovisual Streams in Naturalistic Encoding
Distributed and Intelligent Optimization Research Laboratory（分布式智能优化研究实验室）；Electrical Engineering Department（电气工程系）；Amirkabir University of Technology（阿米尔卡比尔技术大学）
University of Galway（戈尔韦大学）；Technological University Dublin（都柏林理工大学）
Internet memes are characterised by the interspersing of text amongst visual elements. State-of-the-art multimodal meme classifiers do not account for the relative positions of these elements across the two modalities, despite the latent meaning associated with where text and visual elements are placed. Against two meme sentiment classification datasets, we systematically show performance gains from incorporating the spatial position of visual objects, faces, and text clusters extracted from memes. In addition, we also present facial embedding as an impactful enhancement to image representatio…
INRIA（法国国家信息与自动化研究所）；Indian Institute of Information Technology, Allahabad（印度阿拉哈巴德信息技术学院）；Birla Institute of Technology and Science, Pilani（印度皮拉尼贝拉技术科学学院）
Personality computing and affective computing have gained recent interest in many research areas. The datasets for the task generally have multiple modalities like video, audio, language and bio-signals. In this paper, we propose a flexible model for the task which exploits all available data. The task involves complex relations and to avoid using a large model for video processing specifically, we propose the use of behaviour encoding which boosts performance with minimal change to the model. Cross-attention using transformers has become popular in recent times and is utilised for fusion of d…
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）
ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers
Chair of Cyber-Physical Systems（感知系统教授席位）；Institute of Machine Learning and Neural Computation（机器学习与神经计算研究所）；Technical University of Leoben（莱比锡技术大学）；Graz University of Technology（格拉茨技术大学）
StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation
University of Science and Technology of China（科学技术大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究所（TeleAI），中国电信）；Northwestern Polytechnical University（西北工业大学）
A Multimodal RAG Framework for Housing Damage Assessment: Collaborative Optimization of Image Encoding and Policy Vector Retrieval
University of California, Berkeley（加州大学伯克利分校）；Blum Center for Developing Economies（布鲁姆发展中经济体中心）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；McKelvey School of Engineering, Washington University in St. Louis（圣路易斯华盛顿大学麦凯维工程学院）
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Inspired by the fact that different modalities in videos carry complementary information, we propose a Multimodal Semantic Attention Network(MSAN), which is a new encoder-decoder framework incorporating multimodal semantic attributes for video captioning. In the encoding phase, we detect and generate multimodal semantic attributes by formulating it as a multi-label classification problem. Moreover, we add auxiliary classification loss to our model that can obtain more effective visual features and high-level multimodal semantic attribute distributions for sufficient video encoding. In the deco…
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
MAIS, Institute of Automation, Chinese Academy of Sciences, China（MAIS，自动化研究所，中国科学院，中国）；University of Chinese Academy of Sciences, China（中国科学院大学，中国）；Tencent Hunyuan Team, China（腾讯文言团队，中国）
This paper reports the analysis of audio and visual features in predicting the continuous emotion dimensions under the seventh Audio/Visual Emotion Challenge (AVEC 2017), which was done as part of a B.Tech. 2nd year internship project. For visual features we used the HOG (Histogram of Gradients) features, Fisher encodings of SIFT (Scale-Invariant Feature Transform) features based on Gaussian mixture model (GMM) and some pretrained Convolutional Neural Network layers as features; all these extracted for each video clip. For audio features we used the Bag-of-audio-words (BoAW) representation of …
We introduce a novel approach for scanned document representation to perform field extraction. It allows the simultaneous encoding of the textual, visual and layout information in a 3-axis tensor used as an input to a segmentation model. We improve the recent Chargrid and Wordgrid \cite{chargrid} models in several ways, first by taking into account the visual modality, then by boosting its robustness in regards to small datasets while keeping the inference time low. Our approach is tested on public and private document-image datasets, showing higher performances compared to the recent state-of…
Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors
School of Integrated Circuits, Wuxi Vocational College of Science and Technology（集成电路学院，无锡科学技术职业技术学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（先进科技学院，西安交通大学利物浦大学）；Shanghai Gem Science and Technology Co., Ltd.（上海 gems 科技有限公司）；Department of Oncology, Shanghai Medical College, Fudan University（肿瘤科，复旦大学上海医学院）；Department of Medical Ultrasound, Fudan University Shanghai Cancer Center（医学超声科，复旦大学上海癌症中心）
University of Insubria（因苏布里亚大学）；National University of Sciences and Technology（国立科技大学）
This paper proposes a cross-modal retrieval system that leverages on image and text encoding. Most multimodal architectures employ separate networks for each modality to capture the semantic relationship between them. However, in our work image-text encoding can achieve comparable results in terms of cross-modal retrieval without having to use a separate network for each modality. We show that text encodings can capture semantic relationships between multiple modalities. In our knowledge, this work is the first of its kind in terms of employing a single network and fused image-text embedding f…
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
School of Computer Science, Peking University（北京大学计算机学院）；The Chinese University of Hong Kong（香港中文大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding
School of Computer Science（计算机科学学院）；Technology, Huazhong University of Science（科技，华中科技大学）；Li Auto Inc.（Li汽车公司）；Institute of AI for Industries, Chinese Academy of Sciences（产业人工智能研究所，中国科学院）
MoME: Mixture of Multimodal Experts for Cancer Survival Prediction
The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology（香港科技大学）；Tencent YouTu Lab（腾讯优图实验室）；Nanyang Technological University（南洋理工大学）
C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning
State Key Laboratory of Internet of Things for Smart City, University of Macau（物联网智能城市国家重点实验室，澳门大学）；Department of Automation, Shanghai Jiaotong University（上海交通大学自动化系）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；School of Computer Science and Technology, USTC（中国科学技术大学计算机科学与技术学院）；School of Artificial Intelligence and Data Science, USTC（中国科学技术大学人工智能与数据科学学院）
Multimodal imaging and correlative analysis typically require image alignment. Contrastive learning can generate representations of multimodal images, reducing the challenging task of multimodal image registration to a monomodal one. Previously, additional supervision on intermediate layers in contrastive learning has improved biomedical image classification. We evaluate if a similar approach improves representations learned for registration to boost registration performance. We explore three approaches to add contrastive supervision to the latent features of the bottleneck layer in the U-Nets…
MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs
East China Normal University Shanghai China；Centre for Artificial Intelligence；Robotics, Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences Hong Kong China；Department of Computing, The Hong Kong Polytechnic University Hong Kong China；East China Normal University；Robotics, Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences；Department of Computing, The Hong Kong Polytechnic University
Space Applications Centre, ISRO（印度空间研究组织空间应用中心）；Birla Institute of Technology and Science, Pilani（皮拉尼博拉理工学院）
This paper addresses the task of semantic segmentation of orthoimagery using multimodal data e.g. optical RGB, infrared and digital surface model. We propose a deep convolutional neural network architecture termed OrthoSeg for semantic segmentation using multimodal, orthorectified and coregistered data. We also propose a training procedure for supervised training of OrthoSeg. The training procedure complements the inherent architectural characteristics of OrthoSeg for preventing complex co-adaptations of learned features, which may arise due to probable high dimensionality and spatial correlat…
