大型多模态模型（LMMs）近期取得了巨大成功，展现出理解多模态信息及与人类用户交互的强大能力。尽管取得了进展，但在多模态设置中，特别是语音模态中检测高风险交互的挑战，在很大程度上仍未被探索。针对语音模态风险的传统研究主要强调内容（例如，转录所捕获的信息）。然而，在基于语音的交互中，音频中的副语言线索会显著改变话语背后的预期含义。在这项工作中，我们提出了一种语音特定的风险分类体系，涵盖了敌意（恶意讽刺和威胁）、恶意模仿（年龄、性别、种族）和刻板偏见（年龄、性别、种族）下的 8 种风险类别。基于该分类体系，我们创建了一个小规模数据集，用于评估当前 LMMs 检测这些类别风险的能力。我们观察到，即使是最新模型在检测语音中各种副语言特定风险方面仍然无效（例如，Gemini 1.5 Pro 的表现仅略高于随机基准）。警告：本文包含偏见和冒犯性示例。
英文摘要：
Large Multimodal Models (LMMs) have achieved great success recently, demonstrating a strong capability to understand multimodal information and to interact with human users. Despite the progress made, the challenge of detecting high-risk interactions in multimodal settings, and in particular in speech modality, remains largely unexplored. Conventional research on risk for speech modality primarily emphasises the content (e.g., what is captured as transcription). However, in speech-based interactions, paralinguistic cues in audio can significantly alter the intended meaning behind utterances. In this work, we propose a speech-specific risk taxonomy, covering 8 risk categories under hostility (malicious sarcasm and threats), malicious imitation (age, gender, ethnicity), and stereotypical biases (age, gender, ethnicity). Based on the taxonomy, we create a small-scale dataset for evaluating current LMMs capability in detecting these categories of risk. We observe even the latest models remain ineffective to detect various paralinguistic-specific risks in speech (e.g., Gemini 1.5 Pro is performing only slightly above random baseline). Warning: this paper contains biased and offensive examples.
Facebook AI Research（Facebook人工智能研究院）；EHESS（社会科学高等学院）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）
We introduce a new unsupervised task, spoken language modeling: the learning of linguistic representations from raw audio signals without any labels, along with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot metrics probing for the quality of the learned models at 4 linguistic levels: phonetics, lexicon, syntax and semantics. We present the results and analyses of a composite baseline made of the concatenation of three unsupervised systems: self-supervised contrastive representation learning (CPC), clustering (k-means) and language modeling (LSTM or BERT). The langu…
Modern toxic speech detectors are incompetent in recognizing disguised offensive language, such as adversarial attacks that deliberately avoid known toxic lexicons, or manifestations of implicit bias. Building a large annotated dataset for such veiled toxicity can be very expensive. In this work, we propose a framework aimed at fortifying existing toxic speech detectors without a large labeled corpus of veiled toxicity. Just a handful of probing examples are used to surface orders of magnitude more disguised offenses. We augment the toxic speech detector's training data with these discovered o…
ART: The Alternating Reading Task Corpus for Speech Entrainment and Imitation
Italian Institute of Technology（意大利技术研究院）；University of Ferrara（费拉拉大学）；Constantine the Philosopher University in Nitra（尼特拉君士坦丁哲学家大学）；Institute of Informatics, SAS（斯洛伐克科学院信息学研究所）；Aix Marseille University（艾克斯-马赛大学）
NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
Nanjing University（南京大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；The Chinese University of Hong Kong（香港中文大学）；University of Science and Technology Beijing（北京科技大学）；Northwestern Polytechnical University（西北工业大学）；Shanghai Jiao Tong University（上海交通大学）；National Taiwan University（国立台湾大学）
A Modular Taxonomy for Hate Speech Definitions and Its Impact on Zero-Shot LLM Classification Performance
Department of Linguistics, Cognitive Science and Semiotics - Aarhus University（语言学、认知科学与象征学系 - 奥胡斯大学）；GESIS - Leibniz Institute for the Social Sciences（莱布尼茨社会科学研究所）；Heinrich-Heine University Düsseldorf（杜伊斯堡-艾森大学）
Probing for Phonology in Self-Supervised Speech Representations: A Case Study on Accent Perception
Department of Linguistics, University of Florida（语言学系，佛罗里达大学）；Department of English Language and Linguistics, Heinrich Heine University Düsseldorf（英语语言与语言学系，海因里希-海涅大学杜塞尔多夫）
End-to-End automatic speech recognition (ASR) models aim to learn a generalised speech representation to perform recognition. In this domain there is little research to analyse internal representation dependencies and their relationship to modelling approaches. This paper investigates cross-domain language model dependencies within transformer architectures using SVCCA and uses these insights to exploit modelling approaches. It was found that specific neural representations within the transformer layers exhibit correlated behaviour which impacts recognition performance. Altogether, this work p…
Probing experiments investigate the extent to which neural representations make properties -- like part-of-speech -- predictable. One suggests that a representation encodes a property if probing that representation produces higher accuracy than probing a baseline representation like non-contextual word embeddings. Instead of using baselines as a point of comparison, we're interested in measuring information that is contained in the representation but not in the baseline. For example, current methods can detect when a representation is more useful than the word identity (a baseline) for predict…
IIIT-Delhi（德里印度理工学院信息技术学院）；Adobe（奥多比公司）；State University of New York at Buffalo（纽约州立大学布法罗分校）
In recent times, BERT based transformer models have become an inseparable part of the 'tech stack' of text processing models. Similar progress is being observed in the speech domain with a multitude of models observing state-of-the-art results by using audio transformer models to encode speech. This begs the question of what are these audio transformer models learning. Moreover, although the standard methodology is to choose the last layer embedding for any downstream task, but is it the optimal choice? We try to answer these questions for the two recent audio transformer models, Mockingjay an…
University of Augsburg（奥格斯堡大学）；audEERING GmbH（audEERING有限公司）；Imperial College（帝国理工学院）
Large, pre-trained neural networks consisting of self-attention layers (transformers) have recently achieved state-of-the-art results on several speech emotion recognition (SER) datasets. These models are typically pre-trained in self-supervised manner with the goal to improve automatic speech recognition performance -- and thus, to understand linguistic information. In this work, we investigate the extent in which this information is exploited during SER fine-tuning. Using a reproducible methodology based on open-source tools, we synthesise prosodically neutral speech utterances while varying…
UC San Diego（加利福尼亚大学圣迭戈分校）；Georgia Institute of Technology（佐治亚理工学院）
Hate speech has grown significantly on social media, causing serious consequences for victims of all demographics. Despite much attention being paid to characterize and detect discriminatory speech, most work has focused on explicit or overt hate speech, failing to address a more pervasive form based on coded or indirect language. To fill this gap, this work introduces a theoretically-justified taxonomy of implicit hate speech and a benchmark corpus with fine-grained labels for each message and its implication. We present systematic analyses of our dataset using contemporary baselines to detec…
Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American English
Department of English Language and Linguistics, Institute of English and American Studies, Faculty of Arts and Humanities, Heinrich Heine University Düsseldorf, Germany（海因里希·海涅大学杜塞尔多夫分校，人文与艺术学院，英美研究学院，英语语言与语言学系）；Department of Linguistics, University of Florida, United States of America（佛罗里达大学语言学系）
Confident, Calibrated, or Complicit: Probing the Trade-offs between Safety Alignment and Ideological Bias in Language Models in Detecting Hate Speech
School of Physics, Mathematics and Computing（物理、数学与计算学系）；Network Analysis and Social Influence Modelling (NASIM) Lab（网络分析与社会影响建模实验室）；The University of Western Australia（西澳大学）
Speech acts are a way to conceptualize speech as action. This holds true for communication on any platform, including social media platforms such as Twitter. In this paper, we explored speech act recognition on Twitter by treating it as a multi-class classification problem. We created a taxonomy of six speech acts for Twitter and proposed a set of semantic and syntactic features. We trained and tested a logistic regression classifier using a data set of manually labelled tweets. Our method achieved a state-of-the-art performance with an average F1 score of more than $0.70$. We also explored cl…
Institute for Infocomm Research, Agency for Science, Technology and Research（新加坡科技研究局资讯通信研究院）
Embeddings play an important role in end-to-end solutions for multi-modal language processing problems. Although there has been some effort to understand the properties of single-modality embedding spaces, particularly that of text, their cross-modal counterparts are less understood. In this work, we study some intrinsic properties of a joint speech-text embedding space, constructed by minimizing the distance between paired utterance and transcription inputs in a teacher-student model setup, that are informative for several prominent use cases. We found that incorporating automatic speech reco…
