arXiv 2410.066702025-09-08eess.AScs.SD

LS-EEND：基于在线吸引子提取的长时流式端到端说话人日志

LS-EEND: Long-Form Streaming End-to-End Neural Diarization with Online Attractor Extraction

Zhejiang University（浙江大学）；Westlake University（西湖大学）；School of Engineering（工程学院）；Institute of Advanced Technology（先进技术研究所）；Westlake Institute for Advanced Study（西湖研究学院）

Di Liang, Xiaofei Li

备注：: Accepted to IEEE Transactions on Audio, Speech and Language Processing

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出LS-EEND，一种基于在线吸引子提取的长时流式端到端说话人日志方法，采用因果编码器和保留机制，实现线性复杂度的高达8说话人及一小时音频的流式处理，并在多个数据集上取得最先进性能。

摘要：

本文提出了一种逐帧在线/流式端到端说话人日志（EEND）方法，该方法以逐帧输入输出的方式检测说话人活动。所提出的模型主要由一个因果嵌入编码器和一个在线吸引子解码器组成。说话人在基于自注意力的解码器中沿时间和说话人两个维度进行建模，并且逐帧的说话人吸引子分别针对新说话人和现有说话人自动生成和更新。采用保留机制，并特别针对长时日志进行了适配，具有线性时间复杂度。提出了一种多步渐进训练策略，用于在说话人数量和音频长度方面从简单任务逐步学习到困难任务。最后，所提出的模型（称为长时流式EEND，LS-EEND）能够对高达8个且数量灵活的说话人以及非常长（例如一小时）的音频录音执行流式日志。在各种模拟和真实世界数据集上的实验表明：1）在不使用真实说话人活动信息的情况下，所提出的模型在所有数据集上取得了新的最先进在线日志错误率，包括CALLHOME（12.11%）、DIHARD II（27.58%）、DIHARD III（19.61%）和AMI（20.76%）；2）由于逐帧输入输出的处理方式和线性时间复杂度，所提出的模型实现了比对比在线日志模型低数倍的实时因子。

英文摘要：

This work proposes a frame-wise online/streaming end-to-end neural diarization (EEND) method, which detects speaker activities in a frame-in-frame-out fashion. The proposed model mainly consists of a causal embedding encoder and an online attractor decoder. Speakers are modeled in the self-attention-based decoder along both the time and speaker dimensions, and frame-wise speaker attractors are automatically generated and updated for new speakers and existing speakers, respectively. Retention mechanism is employed and especially adapted for long-form diarization with a linear temporal complexity. A multi-step progressive training strategy is proposed for gradually learning from easy tasks to hard tasks in terms of the number of speakers and audio length. Finally, the proposed model (referred to as long-form streaming EEND, LS-EEND) is able to perform streaming diarization for a high (up to 8) and flexible number speakers and very long (say one hour) audio recordings. Experiments on various simulated and real-world datasets show that: 1) when not using oracle speech activity information, the proposed model achieves new state-of-the-art online diarization error rate on all datasets, including CALLHOME (12.11%), DIHARD II (27.58%), DIHARD III (19.61%), and AMI (20.76%); 2) Due to the frame-in-frame-out processing fashion and the linear temporal complexity, the proposed model achieves several times lower real-time-factor than comparison online diarization models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。