While the Self-Attention mechanism in the Transformer model has proven to be effective in many domains, we observe that it is less effective in more diverse settings (e.g. multimodality) due to the varying granularity of each token and the high computational demands of lengthy sequences. To address the challenges, we introduce the Learnable Attention Mask (LAM), strategically designed to globally regulate attention maps and prioritize critical tokens within the sequence. Leveraging the Self-Attention module in a BERT-like transformer network, our approach adeptly captures associations between tokens. The extension of the LAM to a multi-layer version accommodates the varied information aspects embedded at each layer of the Transformer network. Comprehensive experimental validation on various datasets, such as MADv2, QVHighlights, ImageNet 1K, and MSRVTT, demonstrates the efficacy of the LAM, exemplifying its ability to enhance model performance while mitigating redundant computations. This pioneering approach presents a significant advancement in enhancing the understanding of complex scenarios, such as in movie understanding.
Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation
College of Computer and Mathematics, Central South University of Forestry and Technology(中南林业科技大学计算机与数学学院);Department of Computer Science, State University of New York(纽约州立大学计算机科学系)
MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis
National Technical University of Athens(雅典国家技术大学);Institute for Language and Speech Processing, Athena Research Center(雅典娜研究中心语言与语音处理研究所);Behavioral Signal Technologies(行为信号技术公司)
MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents
Peking University(北京大学);HKUST(GZ)(香港科技大学(广州));Tsinghua University(清华大学);The University of Hong Kong(香港大学);Zhejiang University(浙江大学);Huawei Cloud BU(华为云业务部)
SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival
College of Computer Science and Electronic Engineering, Hunan University(湖南大学计算机与通信工程学院);The Second Xiangya Hospital, Central South University(中南大学湘雅二医院)
LUM-ViT: Learnable Under-sampling Mask Vision Transformer for Bandwidth Limited Optical Signal Acquisition
College of Information Science & Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院)
Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所人机混合增强智能全国重点实验室)
View Blind-spot as Inpainting: Self-Supervised Denoising with Mask Guided Residual Convolution
McGill University(麦吉尔大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院)
Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking
Faculty of Information Technology, Brno University of Technology(布尔诺理工大学信息技术学院);Kempelen Institute of Intelligent Technologies(肯佩伦智能技术研究所);George Mason University(乔治梅森大学)
Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing
Great Bay University(大湾区大学);Nanyang Technological University(南洋理工大学);University of Oulu(奥卢大学);Lappeenranta-Lahti University of Technology(拉彭兰塔-拉赫蒂理工大学);South China University of Technology(华南理工大学)
近年来,基于视觉Transformer(ViT)的多模态学习方法被提出来提高人脸反欺骗(FAS)系统的鲁棒性。然而,仍然没有工作来探索的基本性质(\textit{e.g.},模态感知输入,合适的多模态预训练,和有效的微调),在香草ViT多模态FAS。在本文中,我们研究了三个关键因素(即,输入、预训练和微调),用于具有RGB、红外(IR)和深度的多模态FAS。首先,在ViT输入方面,我们发现利用局部特征描述符有利于IR模态上的ViT,而不是RGB或深度模态。其次,在观察的效率直接微调整个或部分ViT,我们设计了一个自适应多模态适配器(AMA),它可以有效地聚合本地多模态功能,同时冻结大多数ViT参数。最后,考虑到任务(FAS与通用对象分类)和模态(多模态与单峰)的差距,ImageNet预训练模型对于多模态FAS任务可能是次优的。为了弥合这些差距,我们提出了模态非对称掩码自编码器(M $^{2}$ A $^{2}$ E),用于多模态FAS自监督预训练,而无需昂贵的注释标签。与先前的模态对称自动编码器相比,所提出的M $^{2}$ A $^{2}$ E能够学习更多内在的任务感知表示并且与模态不可知(例如,单峰、双峰和三峰)下游设置。在多模态FAS基准上进行的单峰(RGB,深度,IR)和多模态(RGB+深度,RGB+IR,深度+IR,RGB+深度+IR)设置的广泛实验证明了所提出的方法的优…
Learnable Shape Prototypes with Occlusion-Geometry-Guided Injection for Amodal Instance Segmentation
School of Mechanical Engineering and Automation, Northeastern University(机械工程与自动化学院,东北大学);School of Information Science and Engineering, Northeastern University(信息科学与工程学院,东北大学)
Out-of-Distribution Detection with Attention Head Masking for Multimodal Document Classification
University of Bristol(布里斯托尔大学);Amazon GenAI(亚马逊生成式人工智能部门);James Silberrad Brown Center for Artificial Intelligence(詹姆斯·西尔伯拉德·布朗人工智能中心);Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学)
Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(人工智能学院、光学和电子学(iOPEN)、西北工业大学);Institute of Artificial Intelligence, China Telecom (TeleAI)(人工智能研究院、中国电信(TeleAI));Fudan University(复旦大学);The University of Hong Kong(香港大学)
MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models
Microsoft Health Futures(微软健康未来);Microsoft Research India(微软研究院印度分院);University of Cambridge(剑桥大学);Cambridge University Hospitals NHS Foundation Trust(剑桥大学医院国民保健信托基金)
Subtractive Modulative Network with Learnable Periodic Activations
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Sweden(电气工程与计算机科学学院,皇家理工学院,瑞典);School of Computer and Communication Sciences, EPFL, Switzerland(计算机与通信科学学院,瑞士联邦理工学院)