PersonaTAB: Predicting Personality Traits using Textual, Acoustic, and Behavioral Cues in Fully-Duplex Speech Dialogs
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University, Suzhou（南京大学（苏州））；National University of Singapore（新加坡国立大学）；School of Data Science（数据科学学院）；School of Intelligence Science and Technology（智能科学与技术学院）；Department of ECE（电子与计算机工程系）
Despite significant progress in neural spoken dialog systems, personality-aware conversation agents -- capable of adapting behavior based on personalities -- remain underexplored due to the absence of personality annotations in speech datasets. We propose a pipeline that preprocesses raw audio recordings to create a dialogue dataset annotated with timestamps, response types, and emotion/sentiment labels. We employ an automatic speech recognition (ASR) system to extract transcripts and timestamps, then generate conversation-level annotations. Leveraging these annotations, we design a system that employs large language models to predict conversational personality. Human evaluators were engaged to identify conversational characteristics and assign personality labels. Our analysis demonstrates that the proposed system achieves stronger alignment with human judgments compared to existing approaches.
Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）；UC Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
University of Pittsburgh（匹兹堡大学）；Adapt Centre, Trinity College Dublin（都柏林圣三一学院适应中心）
Research on human spoken language has shown that speech plays an important role in identifying speaker personality traits. In this work, we propose an approach for identifying speaker personality traits using overlap dynamics in multiparty spoken dialogues. We first define a set of novel features representing the overlap dynamics of each speaker. We then investigate the impact of speaker personality traits on these features using ANOVA tests. We find that features of overlap dynamics significantly vary for speakers with different levels of both Extraversion and Conscientiousness. Finally, we f…
Phoenix-VAD: Streaming Semantic Endpoint Detection for Full-Duplex Speech Interaction
School of Informatics, Xiamen University, China（厦门大学信息学院）；School of Electronic Science and Engineering, Xiamen University, China（厦门大学电子科学与技术学院）；DiDi Global Inc., Beijing, China（滴滴全球有限公司）
NVIDIA（英伟达）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Text normalization (TN) and inverse text normalization (ITN) are essential preprocessing and postprocessing steps for text-to-speech synthesis and automatic speech recognition, respectively. Many methods have been proposed for either TN or ITN, ranging from weighted finite-state transducers to neural networks. Despite their impressive performance, these methods aim to tackle only one of the two tasks but not both. As a result, in a complete spoken dialog system, two separate models for TN and ITN need to be built. This heterogeneity increases the technical complexity of the system, which in tu…
