以人类为中心的多模态融合用于表情包中性别歧视检测：结合眼动追踪、心率和EEG信号
Human-Centered Multimodal Fusion for Sexism Detection in Memes with Eye-Tracking, Heart Rate, and EEG Signals
AI总结：
本文提出了一种结合眼动追踪、心率和EEG信号的多模态融合模型，用于更准确地检测表情包中的性别歧视。
摘要：
自动检测表情包中的性别歧视是一项具有挑战性的任务，由于多模态模糊性、文化细微差别以及幽默提供合理否认的使用。仅内容的模型通常无法捕捉人类感知的复杂性。为了解决这一限制，我们引入并验证了一种以人类为中心的范式，通过补充标准内容特征来增加生理数据。我们创建了一个新的资源，通过记录16名受试者（每项实验8人）在观看EXIST 2025数据集中的3984个表情包时的眼动追踪（ET）、心率（HR）和脑电图（EEG）数据。我们的统计分析揭示了受试者在处理性别歧视与非性别歧视内容时的显著生理差异。性别歧视表情包与更高的认知负荷相关，这体现在更高的注视次数和更长的反应时间，以及在Alpha、Beta和Gamma波段的EEG功率差异，表明更 demanding 的神经处理。基于这些发现，我们提出了一种多模态融合模型，该模型整合了生理信号与从视觉-语言模型（VLM）中提取的丰富文本-视觉特征。我们的最终模型在二元性别歧视检测中实现了0.794的AUC，比强大的VLM基线模型提高了3.4%。融合在细微案例中特别有效，将最具有挑战性的细粒度类别“性别歧视和非性暴力” 的F1分数提高了26.3%。这些结果表明，生理反应提供了一种客观的感知信号，增强了对抗在线性别歧视的自动系统在准确性和人类意识方面的表现。
英文摘要：
The automated detection of sexism in memes is a challenging task due to multimodal ambiguity, cultural nuance, and the use of humor to provide plausible deniability. Content-only models often fail to capture the complexity of human perception. To address this limitation, we introduce and validate a human-centered paradigm that augments standard content features with physiological data. We created a novel resource by recording Eye-Tracking (ET), Heart Rate (HR), and Electroencephalography (EEG) from 16 subjects (8 per experiment) while they viewed 3984 memes from the EXIST 2025 dataset. Our statistical analysis reveals significant physiological differences in how subjects process sexist versus non-sexist content. Sexist memes were associated with higher cognitive load, reflected in increased fixation counts and longer reaction times, as well as differences in EEG spectral power across the Alpha, Beta, and Gamma bands, suggesting more demanding neural processing. Building on these findings, we propose a multimodal fusion model that integrates physiological signals with enriched textual-visual features derived from a Vision-Language Model (VLM). Our final model achieves an AUC of 0.794 in binary sexism detection, a statistically significant 3.4% improvement over a strong VLM-based baseline. The fusion is particularly effective for nuanced cases, boosting the F1-score for the most challenging fine-grained category, Misogyny and Non-Sexual Violence, by 26.3%. These results show that physiological responses provide an objective signal of perception that enhances the accuracy and human-awareness of automated systems for countering online sexism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
