尽管近年来取得了进展，但在工业级语音AI代理部署中，高效且鲁棒的轮流检测仍是一个重大挑战。许多现有系统仅依赖于声学或语义线索，导致准确性不高且稳定性差，而最近尝试赋予大语言模型全双工能力的方法需要昂贵的全双工数据，并带来显著的训练和部署开销，限制了实时性能。在本文中，我们提出了JAL-Turn，一种轻量且高效的语音-only轮流检测框架，采用联合声学-语言建模范式，在其中交叉注意力模块自适应地整合预训练声学表示与语言特征，以支持低延迟预测持 vs 移动状态。通过共享冻结的ASR编码器，JAL-Turn使轮流检测预测能够与语音识别完全并行运行，不引入额外的端到端延迟或计算开销。此外，我们引入了一种可扩展的数据构建管道，自动从大规模真实世界对话语料库中推导出可靠的轮流标签。在公共多语言基准和内部日语客服数据集上的广泛实验表明，JAL-Turn在检测准确性方面始终优于强大的最新基线，同时保持了优越的实时性能。
英文摘要：
Despite recent advances, efficient and robust turn-taking detection remains a significant challenge in industrial-grade Voice AI agent deployments. Many existing systems rely solely on acoustic or semantic cues, leading to suboptimal accuracy and stability, while recent attempts to endow large language models with full-duplex capabilities require costly full-duplex data and incur substantial training and deployment overheads, limiting real-time performance. In this paper, we propose JAL-Turn, a lightweight and efficient speech-only turn-taking framework that adopts a joint acoustic-linguistic modeling paradigm, in which a cross-attention module adaptively integrates pre-trained acoustic representations with linguistic features to support low-latency prediction of hold vs shift states. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with speech recognition, introducing no additional end-to-end latency or computational overhead. In addition, we introduce a scalable data construction pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Extensive experiments on public multilingual benchmarks and an in-house Japanese customer-service dataset show that JAL-Turn consistently outperforms strong state-of-the-art baselines in detection accuracy while maintaining superior real-time performance.
Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）；UC Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems
The Chinese University of Hong Kong（香港中文大学）；AudioLab Hong Kong, Huawei Leibniz Research Center（香港AudioLab，华为Leibniz研究中心）；Nanyang Technological University（南洋理工大学）
NVIDIA（英伟达）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Text normalization (TN) and inverse text normalization (ITN) are essential preprocessing and postprocessing steps for text-to-speech synthesis and automatic speech recognition, respectively. Many methods have been proposed for either TN or ITN, ranging from weighted finite-state transducers to neural networks. Despite their impressive performance, these methods aim to tackle only one of the two tasks but not both. As a result, in a complete spoken dialog system, two separate models for TN and ITN need to be built. This heterogeneity increases the technical complexity of the system, which in tu…
Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the Transcript
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Future Design Lab, Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心未来设计实验室）；University of Virginia（弗吉尼亚大学）
In this paper, we present Duplex Conversation, a multi-turn, multimodal spoken dialogue system that enables telephone-based agents to interact with customers like a human. We use the concept of full-duplex in telecommunication to demonstrate what a human-like interactive experience should be and how to achieve smooth turn-taking through three subtasks: user state detection, backchannel selection, and barge-in detection. Besides, we propose semi-supervised learning with multimodal data augmentation to leverage unlabeled data to increase model generalization. Experimental results on three sub-ta…
Tencent（腾讯）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
PersonaTAB: Predicting Personality Traits using Textual, Acoustic, and Behavioral Cues in Fully-Duplex Speech Dialogs
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University, Suzhou（南京大学（苏州））；National University of Singapore（新加坡国立大学）；School of Data Science（数据科学学院）；School of Intelligence Science and Technology（智能科学与技术学院）；Department of ECE（电子与计算机工程系）
School of Computer Science and Engineering, Northeastern University, Shenyang, China（计算机科学与工程学院，东北大学，沈阳，中国）；NiuTrans Research（NiuTrans研究）；Kunming University of Science and Technology（昆明理工大学）
Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）；Center for Language, Intelligence and Machines, Shenzhen Loop Area Institute, Shenzhen（深圳环智中语言、智能与机器中心）；School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）人工智能学院）
