Death by suicide is the seventh leading death cause worldwide. The recent advancement in Artificial Intelligence (AI), specifically AI applications in image and voice processing, has created a promising opportunity to revolutionize suicide risk assessment. Subsequently, we have witnessed fast-growing literature of research that applies AI to extract audiovisual non-verbal cues for mental illness assessment. However, the majority of the recent works focus on depression, despite the evident difference between depression symptoms and suicidal behavior and non-verbal cues. This paper reviews recent works that study suicide ideation and suicide behavior detection through audiovisual feature analysis, mainly suicidal voice/speech acoustic features analysis and suicidal visual cues. Automatic suicide assessment is a promising research direction that is still in the early stages. Accordingly, there is a lack of large datasets that can be used to train machine learning and deep learning models proven to be effective in other, similar tasks.
Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty
Faculty of Education and Psychology, University of Oulu, Finland（奥卢大学教育与心理学学院，芬兰）；Center for Machine Vision and Signal Analysis, University of Oulu, Finland（奥卢大学机器视觉与信号分析中心，芬兰）
Xi’an University of Posts and Telecommunications（西安邮电大学）；University of Chinese Academy of Sciences（中国科学院大学）；Aalto University（阿尔托大学）；Northwest University（西北大学）；Northwestern Polytechnical University（西北工业大学）
With the acceleration of the pace of work and life, people have to face more and more pressure, which increases the possibility of suffering from depression. However, many patients may fail to get a timely diagnosis due to the serious imbalance in the doctor-patient ratio in the world. Promisingly, physiological and psychological studies have indicated some differences in speech and facial expression between patients with depression and healthy individuals. Consequently, to improve current medical care, many scholars have used deep learning to extract a representation of depression cues in aud…
Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；National Taiwan University（台湾大学）；Institute of Linguistics, Academia Sinica（中央研究院语言学研究所）
Speech perception is key to verbal communication. For people with hearing loss, the capability to recognize speech is restricted, particularly in a noisy environment or the situations without visual cues, such as lip-reading unavailable via phone call. This study aimed to understand the improvement of vocoded speech intelligibility in cochlear implant (CI) simulation through two potential methods: Speech Enhancement (SE) and Audiovisual Integration. A fully convolutional neural network (FCN) using an intelligibility-oriented objective function was recently proposed and proven to effectively fa…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
We introduce an audiovisual method for long-range text-to-video retrieval. Unlike previous approaches designed for short video retrieval (e.g., 5-15 seconds in duration), our approach aims to retrieve minute-long videos that capture complex human actions. One challenge of standard video-only approaches is the large computational cost associated with processing hundreds of densely extracted frames from such long videos. To address this issue, we propose to replace parts of the video with compact audio cues that succinctly summarize dynamic audio events and are cheap to process. Our method, name…
Generative artificial intelligence models can be a valuable aid to music composition and live performance, both to aid the professional musician and to help democratize the music creation process for hobbyists. Here we present a novel method that, given an English word or phrase, generates 2 compasses of a 4-piece drum pattern that embodies the "mood" of the given language cue, or that could be used for an audiovisual scene described by the language cue. We envision this tool as composition aid for electronic music and audiovisual soundtrack production, or an improvisation tool for live perfor…
Data fusion plays an important role in many technical applications that require efficient processing of multimodal sensory observations. A prominent example is audiovisual signal processing, which has gained increasing attention in automatic speech recognition, speaker localization and related tasks. If appropriately combined with acoustic information, additional visual cues can help to improve the performance in these applications, especially under adverse acoustic conditions. A dynamic weighting of acoustic and visual streams based on instantaneous sensor reliability measures is an efficient…
AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues
UC Berkeley（加州大学伯克利分校）；South China University of Technology（华南理工大学）；Zhejiang University（浙江大学）；National Taiwan University（台湾大学）；University of Southern California（美国南加州大学）
Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings
Speech and Hearing (SPandH) group, School of Computer Science, The University of Sheffield, Sheffield, United Kingdom（语音与听力（SPandH）小组，计算机科学学院，谢菲尔德大学，谢菲尔德，英国）；South Westphalia University of Applied Sciences, Iserlohn, Germany（西南弗劳恩霍夫应用科学大学，伊塞尔洛恩，德国）
Neural Tracking of Sustained Attention, Attention Switching, and Natural Conversation in Audiovisual Environments using Mobile EEG
Dept. of Electrical Engineering, Linköping University（电子工程系，利厄普堡大学）；Centre for Mathematical Sciences, Lund University（数学科学中心，卢德大学）；Eriksholm Research Centre, Oticon A/S（埃里克霍姆研究中心，Oticon A/S）
Ensembling Synchronisation-based and Face-Voice Association Paradigms for Robust Active Speaker Detection in Egocentric Recordings
Speech and Hearing (SPandH), School of Computer Science, The University of Sheffield, UK（语音与听力（SPandH）、计算机科学学院、谢菲尔德大学、英国）；South Westphalia University of Applied Sciences, Iserlohn, Germany（西南弗兰肯应用科学大学、伊塞尔洛恩、德国）
