arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.21050cs.SD
收藏

ERM-MinMaxGAP:多语言多模态语音-LLM情感识别中的性别偏见基准测试与缓解

ERM-MinMaxGAP: Benchmarking and Mitigating Gender Bias in Multilingual Multimodal Speech-LLM Emotion Recognition

Kyoto University, Japan(京都大学,日本);Agency for Science, Technology, and Research (A*STAR), Singapore(科技研究局(A*STAR),新加坡)

Zi Haur Pang, Xiaoxue Gao, Tatsuya Kawahara, Nancy F. Chen

AI总结:

针对多语言语音大模型在情感识别中的性别偏见问题,提出基于MELD-ST的多语言多模态基准,并设计ERM-MinMaxGAP训练目标,通过自适应公平权重和MinMaxGAP正则化器,在英日德三种语言上提升性能并缩小性别差距。

摘要:

语音情感识别(SER)系统可能表现出与性别相关的性能差异,但这种偏见如何在跨语言和跨模态的多语言语音大模型中体现尚不清楚。我们引入了一个基于MELD-ST的新型多语言多模态基准,涵盖英语、日语和德语,以量化特定语言的SER性能和性别差距。我们发现偏见强烈依赖于语言,并且多模态融合并不能可靠地提高公平性。为了解决这些问题,我们提出了ERM-MinMaxGAP,一种公平性感知的训练目标,它通过提出的自适应公平权重机制和一种新颖的MinMaxGAP正则化器(针对每种语言和模态内的最大男女损失差距)来增强经验风险最小化(ERM)。基于Qwen2-Audio骨干网络,我们的ERM-MinMaxGAP方法在单模态和多模态设置下分别将多语言SER性能提高了5.5%和5.0%,同时将整体性别偏见差距减少了0.1%和1.4%。

英文摘要:

Speech emotion recognition (SER) systems can exhibit gender-related performance disparities, but how such bias manifests in multilingual speech LLMs across languages and modalities is unclear. We introduce a novel multilingual, multimodal benchmark built on MELD-ST, spanning English, Japanese, and German, to quantify language-specific SER performance and gender gaps. We find bias is strongly language-dependent, and multimodal fusion does not reliably improve fairness. To address these, we propose ERM-MinMaxGAP, a fairness-informed training objective, which augments empirical risk minimization (ERM) with a proposed adaptive fairness weight mechanism and a novel MinMaxGAP regularizer on the maximum male-female loss gap within each language and modality. Building upon the Qwen2-Audio backbone, our ERM-MinMaxGAP approach improves multilingual SER performance by 5.5% and 5.0% while reducing the overall gender bias gap by 0.1% and 1.4% in the unimodal and multimodal settings, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑