MAiVAR: Multimodal Audio-Image and Video Action Recognizer

MAiVAR-T：基于Transformer的多模态音频-图像与视频动作识别器

MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers

本文提出MAiVAR-T模型，通过将音频表示转换到图像域并与视频模态融合，提升多模态人类动作识别性能，实验证明其优于仅用单一模态的现有方法。

Students taught by multimodal teachers are superior action recognizers

The focal point of egocentric video understanding is modelling hand-object interactions. Standard models -- CNNs, Vision Transformers, etc. -- which receive RGB frames as input perform well, however, their performance improves further by employing additional modalities such as object detections, optical flow, audio, etc. as input. The added complexity of the required modality-specific modules, on the other hand, makes these models impractical for deployment. The goal of this work is to retain the performance of such multimodal approaches, while using only the RGB images as input at inference t…

Day2Dark：超越静默日光场景的伪监督行为识别

Day2Dark: Pseudo-Supervised Activity Recognition beyond Silent Daylight

针对黑暗环境下行为识别性能差的问题，提出伪监督学习方案和黑暗自适应音视频识别器，利用无标注黑暗视频和光照不变的音频信息提升识别效果，在多个数据集上表现优于现有方法。

Audio-Adaptive Activity Recognition Across Video Domains

This paper strives for activity recognition under domain shift, for example caused by change of scenery or camera viewpoint. The leading approaches reduce the shift in activity appearance by adversarial training and self-supervised learning. Different from these vision-focused works we leverage activity sounds for domain adaptation as they have less variance across domains and can reliably indicate which activities are not happening. We propose an audio-adaptive encoder and associated learning methods that discriminatively adjust the visual feature representation as well as addressing shifts i…