AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；China Tower Corporation Limited（中国铁塔股份有限公司）
Recently, transformer-based models have demonstrated remarkable performance on audio-visual segmentation (AVS) tasks. However, their expensive computational cost makes real-time inference impractical. By characterizing attention maps of the network, we identify two key obstacles in AVS models: 1) attention dissipation, corresponding to the over-concentrated attention weights by Softmax within restricted frames, and 2) inefficient, burdensome transformer decoder, caused by narrow focus patterns in early stages. In this paper, we introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation transformer that achieves fast, efficient and light-weight simultaneously. Our model leverages an efficient prompt query generator to correct the behaviour of cross-attention. Additionally, we propose ELF decoder to bring greater efficiency by facilitating convolutions suitable for local features to reduce computational burdens. Extensive experiments demonstrate that our AVESFormer significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3 and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an excellent trade-off between performance and speed. Code can be found at https://github.com/MarkXCloud/AVESFormer.git.
Hefei University of Technology（合肥工业大学）；Sensetime Research（商汤科技研究院）；University of Oxford（牛津大学）；Beihang University（北京航空航天大学）；Australian National University（澳大利亚国立大学）；Nvidia（英伟达）；University of Hong Kong（香港大学）；Shanghai AI Lab（上海人工智能实验室）
We propose a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark, i.e., AVSBench, providing pixel-wise annotations for sounding objects in audible videos. It contains three subsets: AVSBench-object (Single-source subset, Multi-sources subset) and AVSBench-semantic (Semantic-labels subset). Accordingly, three settings are studied: 1) semi-supervised audio-visual segmentation with a single soun…
Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机学院）；Alibaba Group（阿里巴巴集团）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
CPM: Class-conditional Prompting Machine for Audio-visual Segmentation
Australian Institute for Machine Learning（澳大利亚机器学习研究院）；University of Adelaide（阿德莱德大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Surrey（萨里大学）
Can Textual Semantics Mitigate Sounding Object Segmentation Preference?
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Edinburgh（爱丁堡大学）
TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models
Northwestern Polytechnical University（西北工业大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合国家科学中心人工智能研究院）
Complementary and Contrastive Learning for Audio-Visual Segmentation
School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）；School of Future Technology and the School of Artificial Intelligence, Dalian University of Technology（未来技术学院和人工智能学院，大连理工大学）
Visual objects often have acoustic signatures that are naturally synchronized with them in audio-bearing video recordings. For this project, we explore the multimodal feature aggregation for video instance segmentation task, in which we integrate audio features into our video segmentation model to conduct an audio-visual learning scheme. Our method is based on existing video instance segmentation method which leverages rich contextual information across video frames. Since this is the first attempt to investigate the audio-visual instance segmentation, a novel dataset, including 20 vocal class…
Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment
The University of Queensland（昆士兰大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）；Matrix Verse AI（矩阵 Verse 人工智能公司）；CSIRO Data61（联邦科学与工业研究组织 Data61 部门）；Macau University of Science and Technology（澳门科技大学）
AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation
School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）；School of Innovation and Entrepreneurship, Dalian University of Technology（大连理工大学创新创业学院）；School of Artificial Intelligence, Dalian University of Technology（大连理工大学人工智能学院）
Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation
Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究院）；St Vincent’s Hospital Melbourne（墨尔本圣文森特医院）；Centre for Vision, Speech and Signal Processing, University of Surrey（萨里大学视觉、语音与信号处理中心）
College of Informatics, Huazhong Agricultural University, Wuhan, China（华中农业大学信息学院）；School of Computing, National University of Singapore, Singapore（新加坡国立大学计算机学院）；School of Computer Science, Adelaide University, Australia（阿德莱德大学计算机科学学院）；School of Engineering, University of Warwick, Coventry, UK（沃里克大学工程学院）；Zhejiang Yuexiu University, Shaoxing, China（浙江越秀大学）
Hefei University of Technology（合肥工业大学）；SenseTime Research（商汤科技研究院）；The University of Hong Kong（香港大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Australian National University（澳大利亚国立大学）；Beihang University（北京航空航天大学）；NVIDIA（英伟达）
We propose to explore a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise annotations for the sounding objects in audible videos. Two settings are studied with this benchmark: 1) semi-supervised audio-visual segmentation with a single sound source and 2) fully-supervised audio-visual segmentation with multiple sound sources. To deal with the AVS problem, we …
MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this paper we develop a neural network model for visual object segmentation and sound source separation that learns from natural videos through self-supervision. The model is an extension of recently proposed work that maps image pixels to sounds. Here, we introduce a learning approach to disentangle concepts in the neural networks, and assign semantic categories to network feature channels to enable independent image segment…
Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics
The University of Queensland（昆士兰大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）；Matrix Verse AI；CSIRO Data61（联邦科学与工业研究组织数据61部）；Macau University of Science and Technology（澳门科技大学）
AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting
Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Australian Institute for Machine Learning, Adelaide University（阿德莱德大学人工智能研究所）；Stanford University（斯坦福大学）；University of Central Florida（佛罗里达中央大学）；University of Surrey（萨里大学）
Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics
University of Technology Sydney（悉尼科技大学）；The University of Queensland（昆士兰大学）；Futureverse；The Hong Kong University of Science and Technology（香港科技大学）；CSIRO DATA61（联邦科学与工业研究组织数据61研究所）；Netease Fuxi AI Lab（网易伏羲人工智能实验室）
Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation
University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Engineering Research Center of Next-Generation Search and Recommendation（下一代搜索与推荐工程研究中心）
Over the last decade, online lecture videos have become increasingly popular and have experienced a meteoric rise during the pandemic. However, video-language research has primarily focused on instructional videos or movies, and tools to help students navigate the growing online lectures are lacking. Our first contribution is to facilitate research in the educational domain, by introducing AVLectures, a large-scale dataset consisting of 86 courses with over 2,350 lectures covering various STEM subjects. Each course contains video lectures, transcripts, OCR outputs for lecture frames, and optio…
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
School of Intelligent Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
