arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22710cs.SDcs.AIcs.HC
收藏

相同词语,不同判断:偏好如何在不同模态间变化

Same Words, Different Judgments: How Preferences Vary Across Modalities

Department of Computer Science and Engineering(计算机科学与工程系);University of California San Diego(加州大学圣地亚哥分校);Department of Psychology(心理学系)

Aaron Broukhim, Nadir Weibel, Eshin Jolly

AI总结:

研究探讨了文本与语音模态在偏好标注中的差异,发现需9名评阅者才能达到较高一致性,且语音评阅者在决策阈值、长度偏差和用户导向标准上存在显著差异,合成评分可有效预测评阅者一致性,强调音频偏好数据评估需模态特定设计。

摘要:

基于偏好强化学习(PbRL)是将AI系统对齐人类偏好的主流框架。然而,此类数据的评估协议最初为文本设计,尚未在语音中验证。本文首次提出基于ICC的受控跨模态研究,比较100个提示中相同语义内容的文本和音频评估。研究显示,实现任一模态内良好一致性(ICC(2,k)≈0.80)需约9名评阅者。同时,模态间在人们如何报告偏好上存在显著差异:音频评阅者表现出更窄的决策阈值、减少的长度偏差和更用户导向的评估标准,跨模态一致性接近偶然。研究证明合成评分可有效预测评阅者一致性,从而作为刺激选择的早期信号和人类注释的代理。这些发现表明,音频偏好数据的评估协议需要模态特定设计,而非直接从文本适应。

英文摘要:

Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first intraclass correlation coefficient (ICC)-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving \textit{good} (audio) to \textit{near good} (text) agreement within either modality (ICC(2,$k$) $\approx$ .81 audio, .74 text) requires $\sim$9 raters and modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text. Code, data, and the audio corpus are available at: https://huggingface.co/datasets/sciminds/same-words-different-judgements

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑