本文提出了一种用于情感智能的统一口语语言模型，通过一种称为注入情感归因思维（IEAT）的新数据构建策略进行增强。IEAT将用户的情感状态及其潜在原因纳入模型的内部推理过程，使情感感知推理得以内化而非作为显式监督。模型采用两阶段渐进策略进行训练。第一阶段通过自蒸馏进行语音-文本对齐和情感属性建模，第二阶段进行端到端的跨模态联合优化，以确保文本和口语情感表达之间的一致性。在Human-like Spoken Dialogue Systems Challenge（HumDial）情感智能基准测试中，实验表明，所提出的方法在情感轨迹建模、情感推理和共情响应生成方面均在基于LLM和人工评估中取得领先表现。
英文摘要：
This paper presents a unified spoken language model for emotional intelligence, enhanced by a novel data construction strategy termed Injected Emotional-Attribution Thinking (IEAT). IEAT incorporates user emotional states and their underlying causes into the model's internal reasoning process, enabling emotion-aware reasoning to be internalized rather than treated as explicit supervision. The model is trained with a two-stage progressive strategy. The first stage performs speech-text alignment and emotional attribute modeling via self-distillation, while the second stage conducts end-to-end cross-modal joint optimization to ensure consistency between textual and spoken emotional expressions. Experiments on the Human-like Spoken Dialogue Systems Challenge (HumDial) Emotional Intelligence benchmark demonstrate that the proposed approach achieves top-ranked performance across emotional trajectory modeling, emotional reasoning, and empathetic response generation under both LLM-based and human evaluations.
Low Resource Multimodal Translation of Nepali Spoken Words into Emotion-Conditioned Sign Language Avatars
Center for Human Mobility and Communications, Prateek Innovations（普拉蒂克创新公司人类移动与通信中心）；Sunway International Business School, Birmingham City University（双威国际商学院，伯明翰城市大学）
Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术大学人工智能研究院）；School of Data Science, Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；School of Artificial Intelligence, Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）人工智能学院）
The advances in language-based Artificial Intelligence (AI) technologies applied to build educational applications can present AI for social-good opportunities with a broader positive impact. Across many disciplines, enhancing the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages. Conversational AI systems have started maturing to a point where they could play a significant role in helping students learn fundamental math concepts. This work presents a task-oriented Spoken Dialogue System (SDS) built to support play-based learni…
Numediart Institute（Numediart研究所）；University of Mons（蒙斯大学）
During the last decade, the applications of signal processing have drastically improved with deep learning. However areas of affecting computing such as emotional speech synthesis or emotion recognition from spoken language remains challenging. In this paper, we investigate the use of a neural Automatic Speech Recognition (ASR) as a feature extractor for emotion recognition. We show that these features outperform the eGeMAPS feature set to predict the valence and arousal emotional dimensions, which means that the audio-to-text mapping learning by the ASR system contain information related to t…
Institute of Artificial Intelligence Research, Qihoo of Beijing Science and Technology Co. Ltd.（北京奇虎科技有限公司人工智能研究院）
Historically lower-level tasks such as automatic speech recognition (ASR) and speaker identification are the main focus in the speech field. Interest has been growing in higher-level spoken language understanding (SLU) tasks recently, like sentiment analysis (SA). However, improving performances on SLU tasks remains a big challenge. Basically, there are two main methods for SLU tasks: (1) Two-stage method, which uses a speech model to transfer speech to text, then uses a language model to get the results of downstream tasks; (2) One-stage method, which just fine-tunes a pre-trained speech mode…
South-West University "Neofit Rilski"（西南大学“Neofit Rilski”）
The report presents the process of planning, designing and the development of a database of spoken children's speech whose native language is Bulgarian. The proposed model is designed for children between the age of 4 and 6 without speech disorders, and reflects their specific capabilities. At this age most children cannot read, there is no sustained concentration, they are emotional, etc. The aim is to unite all the media information accompanying the recording and processing of spoken speech, thereby to facilitate the work of researchers in the field of speech recognition. This database will …
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Changi General Hospital（樟宜综合医院）
Data for human-human spoken dialogues for research and development are currently very limited in quantity, variety, and sources; such data are even scarcer in healthcare. In this work, we investigate fast prototyping of a dialogue comprehension system by leveraging on minimal nurse-to-patient conversations. We propose a framework inspired by nurse-initiated clinical symptom monitoring conversations to construct a simulated human-human dialogue dataset, embodying linguistic characteristics of spoken interactions like thinking aloud, self-contradiction, and topic drift. We then adopt an establis…
When a human communicates with a machine using natural language on the web and online, how can it understand the human's intention and semantic context of their talk? This is an important AI task as it enables the machine to construct a sensible answer or perform a useful action for the human. Meaning is represented at the sentence level, identification of which is known as intent detection, and at the word level, a labelling task called slot filling. This dual-level joint task requires innovative thinking about natural language and deep learning network design, and as a result, many approache…
Simultaneous machine translation attempts to translate a source sentence before it is finished being spoken, with applications to translation of spoken language for live streaming and conversation. Since simultaneous systems trade quality to reduce latency, having an effective and interpretable latency metric is crucial. We introduce a variant of the recently proposed Average Lagging (AL) metric, which we call Differentiable Average Lagging (DAL). It distinguishes itself by being differentiable and internally consistent to its underlying mathematical model.
EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
Tianjin University（天津大学）；Fuzhou University（福州大学）；Shanghai Jiaotong University（上海交通大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术研究所）
Multilingual spoken dialogue systems have gained prominence in the recent past necessitating the requirement for a front-end Language Identification (LID) system. Most of the existing LID systems rely on modeling the language discriminative information from low-level acoustic features. Due to the variabilities of speech (speaker and emotional variabilities, etc.), large-scale LID systems developed using low-level acoustic features suffer from a degradation in the performance. In this approach, we have attempted to model the higher level language discriminative phonotactic information for devel…
Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms
University of Information Technology & Sciences（信息技术与科学大学）；Southeast University（东南大学）；Endicott College, Woosong University（又松大学恩迪科特学院）；Daffodil International University（水仙国际大学）；Jahangirnagar University（贾汉吉尔纳加尔大学）
EmoTale: An Enacted Speech-emotion Dataset in Danish
University Grenoble Alpes, CNRS, Grenoble INP, LIG（格陵兰大学、CNRS、格陵兰INP、LIG）；Dept. of Applied Mathematics and Computer Science, Technical University of Denmark（应用数学和计算机科学系，丹麦技术大学）
Recognition of social signals, from human facial expressions or prosody of speech, is a popular research topic in human-robot interaction studies. There is also a long line of research in the spoken dialogue community that investigates user satisfaction in relation to dialogue characteristics. However, very little research relates a combination of multimodal social signals and language features detected during spoken face-to-face human-robot interaction to the resulting user perception of a robot. In this paper we show how different emotional facial expressions of human users, in combination w…
Spoken language proficiency is critically important for children's growth and personal development. Due to the limited and imbalanced educational resources in China, elementary students barely have chances to improve their oral language skills in classes. Verbal fluency tasks (VFTs) were invented to let the students practice their spoken language proficiency after school. VFTs are simple but concrete math related questions that ask students to not only report answers but speak out the entire thinking process. In spite of the great success of VFTs, they bring a heavy grading burden to elementar…
Thousands of short stories and articles are being written in many different languages all around the world today. Bengali, or Bangla, is the second highest spoken language in India after Hindi and is the national language of the country of Bangladesh. This work reports in detail the creation of Anubhuti -- the first and largest text corpus for analyzing emotions expressed by writers of Bengali short stories. We explain the data collection methods, the manual annotation process and the resulting high inter-annotator agreement of the dataset due to the linguistic expertise of the annotators and …
National Computer System Engineering Research Institute of China（中国国家计算机系统工程研究所）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Emotion Recognition in Conversations (ERC) has gained increasing attention for developing empathetic machines. Recently, many approaches have been devoted to perceiving conversational context by deep learning models. However, these approaches are insufficient in understanding the context due to lacking the ability to extract and integrate emotional clues. In this work, we propose novel Contextual Reasoning Networks (DialogueCRN) to fully understand the conversational context from a cognitive perspective. Inspired by the Cognitive Theory of Emotion, we design multi-turn reasoning modules to ext…
Goldsmiths College, University of London（伦敦大学金史密斯学院）；US Department of State（美国国务院）
"Natural Language," whether spoken and attended to by humans, or processed and generated by computers, requires networked structures that reflect creative processes in semantic, syntactic, phonetic, linguistic, social, emotional, and cultural modules. Being able to produce novel and useful behavior following repeated practice gets to the root of both artificial intelligence and human language. This paper investigates the modalities involved in language-like applications that computers -- and programmers -- engage with, and aims to fine tune the questions we ask to better account for context, s…
Speech-based Psychological Crisis Assessment using LLMs
Tsinghua University（清华大学）；Peking University Huilongguan Clinical Medical School（北京大学回龙guan临床医学院）；WHO Collaborating Centre for Research and Training in Suicide Prevention（世界卫生组织自杀预防研究与培训协作中心）
