大语言模型（LLM）近期在视听语音识别（AVSR）领域展现出强劲潜力，但其高计算需求以及对token粒度的敏感性限制了其在资源受限场景下的实用性。Token压缩方法能够降低推理成本，但这类方法需要预先固定压缩率，且仅能输出单一固定长度的结果，无法在推理时灵活平衡信息密度与效率。套娃表示学习（Matryoshka representation learning, MRL）通过让单个模型支持多种token粒度运行解决了这一问题，可动态调整压缩率。然而，当前基于MRL的方法在训练时独立处理每个尺度，限制了跨尺度泛化能力、高压缩比下的鲁棒性以及可解释性。为克服这些局限，我们提出MoME（Mixture of Matryoshka Experts，套娃专家混合模型）——一种将稀疏专家混合（Mixture-of-Experts, MoE）集成到基于MRL的AVSR大语言模型中的新型框架。MoME为冻结的LLM增设top-k路由专家与共享专家，可实现跨尺度与跨模态的动态容量分配；共享路由器可促进不同粒度下专家激活的一致性，让压缩序列能够从更低压缩率下学习到的表示中获益。在LRS2和LRS3数据集上的实验表明，MoME在AVSR、自动语音识别（ASR）和视觉语音识别（VSR）任务上均达到了当前最优性能，同时所需参数量显著更少，且在噪声环境下仍能保持鲁棒性。MoME融合了MRL的自适应性与MoE的高效性，为资源感知型语音识别提供了一种可扩展且具备可解释性的解决方案。
英文摘要：
Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-constrained settings. Token compression methods can reduce inference cost, but they require fixing a compression rate in advance and produce a single fixed-length output, offering no flexibility to balance information density and efficiency at inference time. Matryoshka representation learning (MRL) addresses this by enabling a single model to operate across multiple token granularities, allowing compression rates to be adjusted dynamically. However, current MRL-based methods treat each scale independently during training, limiting cross-scale generalization, robustness at high compression, and interpretability. To overcome these limitations, we propose MoME (Mixture of Matryoshka Experts), a novel framework that integrates sparse Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen LLM with top-k routed and shared experts, allowing dynamic capacity allocation across scales and modalities. A shared router promotes consistent expert activation across granularities, enabling compressed sequences to benefit from representations learned at lower compression. Experiments on LRS2 and LRS3 demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR, and VSR tasks, while requiring significantly fewer parameters and maintaining robustness under noise. MoME unifies the adaptability of MRL with the efficiency of MoE, offering a scalable and interpretable solution for resource-aware speech recognition.
When people observe events, they are able to abstract key information and build concise summaries of what is happening. These summaries include contextual and semantic information describing the important high-level details (what, where, who and how) of the observed event and exclude background information that is deemed unimportant to the observer. With this in mind, the descriptions people generate for videos of different dynamic events can greatly improve our understanding of the key information of interest in each video. These descriptions can be captured in captions that provide expanded …
EDTC: enhance depth of text comprehension in automated audio captioning
School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications（重庆邮电大学通信与信息工程学院）；Department of Intelligent Science, Xi’an Jiaotong-Liverpool University（西交利物浦大学智能科学系）
University of Science and Technology of China（中国科学技术大学）；iFLYTEK Research, iFLYTEK Co. Ltd.（科大讯飞股份有限公司讯飞研究院）
In this work, we present a novel method, named AV2vec, for learning audio-visual speech representations by multimodal self-distillation. AV2vec has a student and a teacher module, in which the student performs a masked latent feature regression task using the multimodal target features generated online by the teacher. The parameters of the teacher model are a momentum update of the student. Since our target features are generated online, AV2vec needs no iteration step like AV-HuBERT and the total training time cost is reduced to less than one-fifth. We further propose AV2vec-MLM in this study,…
Carnegie Mellon University（卡内基梅隆大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video. Since collecting ground-truth annotations of sounding objects can be costly, a plethora of weakly-supervised localization methods that can learn from datasets with no bounding-box annotations have been proposed in recent years, by leveraging the natural co-occurrence of audio and visual signals. Despite significant interest, popular evaluation protocols have two major flaws. First, they allow for the use of a fully annotated dataset to perform early stopping, thus signif…
The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Facebook AI Research（脸书人工智能研究院）
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical. We propose a framework for efficient action recognition in untrimmed video that uses audio as a preview mechanism to eliminate both short-term and long-term visual redundancies. First, we devise an ImgAud2Vid framework that hallucinates clip-level features by distilling from lighter modalities---a single frame and its accompanying audio---reducing short-term temporal redundancy for efficient clip-level recognition. Second, building on ImgAud2Vid, we further propose ImgAud-Skimming, an a…
MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues
The Hong Kong Polytechnic University（香港理工大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Under noisy conditions, automatic speech recognition (ASR) can greatly benefit from the addition of visual signals coming from a video of the speaker's face. However, when multiple candidate speakers are visible this traditionally requires solving a separate problem, namely active speaker detection (ASD), which entails selecting at each moment in time which of the visible faces corresponds to the audio. Recent work has shown that we can solve both problems simultaneously by employing an attention mechanism over the competing video tracks of the speakers' faces, at the cost of sacrificing some …
Difficulties in following speech due to loud background sounds are common in broadcasting. Object-based audio, e.g., MPEG-H Audio solves this problem by providing a user-adjustable speech level. While object-based audio is gaining momentum, transitioning to it requires time and effort. Also, lots of content exists, produced and archived outside the object-based workflows. To address this, Fraunhofer IIS has developed a deep-learning solution called Dialog+, capable of enabling speech level personalization also for content with only the final audio tracks available. This paper reports on public…
