语音事件检测器：用于丰富语音转录的非言语发声与背景声音的带时间戳检测
Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription
AI总结：
本研究提出580万参数的卷积-Transformer语音事件检测器，可检测54类非言语发声与背景声，经多数据集训练调优后，在CPU上运行速度为实时的39倍，能丰富语音转录内容。
摘要：
自动语音识别仅输出文字，会丢失录音中的非言语内容：说话者的呼吸、咳嗽、叹息、笑声，以及周围的喇叭声、音乐或犬吠声。我们描述了Speech Event Detector，这是一个拥有580万参数的卷积-Transformer模型，每40毫秒输出54个事件类别的概率（20种说话者发声和34种背景声音），同时配有将这些概率转换为带时间戳事件的后处理，以及将这些事件合并到Whisper转录文本中的流程。该模型在来自7个公共数据集的材料上进行训练，这些数据集的标签形式互不兼容：单个检测区域、带时间戳的自由文本标签、时间性强的本体标签、嵌入转录文本的符号、孤立录音和环境录音。我们使用基于规则的标签映射统一这些标签，通过CTC强制对齐恢复嵌入转录文本的标签的时间戳，定位了8620个声音中的6360个，并将约168小时的真实标注训练录音与合成混合数据及真实背景噪声增强相结合。针对两个操作配置文件的每类阈值在真实保留录音上进行了调优，性能通过双重交叉验证进行评估。在来自4个数据集的完整保留录音上，考虑至少有10个保留事件的48个类别，敏感配置文件平均检测到一个类别49%的事件，覆盖所有9367个事件的67%，宏观精度为0.30；平衡配置文件在精度为0.36时达到宏观召回率0.37，宏观事件F1分数分别为0.32和0.33。该检测器在8个CPU线程上的运行速度约为实时速度的39倍。
英文摘要：
Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them. We describe Speech Event Detector, a 5.8-million-parameter convolutional-Transformer model that outputs probabilities every 40 ms for 54 event classes (20 speaker vocalizations and 34 background sounds), together with post-processing that converts these probabilities into timestamped events and a procedure for merging the events into a Whisper transcript. The model is trained on material drawn from seven public datasets whose labels come in incompatible forms: single detected regions, free-text tags with timestamps, temporally strong ontology labels, symbols embedded in transcripts, isolated recordings and environmental recordings. We unify them using rule-based label mapping, recover timestamps for transcript-embedded labels through CTC forced alignment, locating 6,360 of 8,620 sounds, and combine about 168 hours of real annotated training recordings with synthetic mixtures and augmentation using real background noise. Per-class thresholds for two operating profiles are tuned on real held-out recordings, and performance is estimated using two-fold cross-validation. On whole held-out recordings from four datasets, considering the 48 classes with at least ten held-out events, the sensitive profile detects an average of 49 percent of a class's events, covering 67 percent of all 9,367 events, at a macro precision of 0.30. The balanced profile achieves a macro recall of 0.37 at a precision of 0.36, with macro event F1 scores of 0.32 and 0.33, respectively. The detector runs about 39 times faster than real time on eight CPU threads.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
