注意差距！大型音频模型的静态与交互式评估
Mind the Gap! Static and Interactive Evaluations of Large Audio Models
AI总结：
本研究通过收集484名参与者的7500次交互，提出交互式评估大型音频模型的方法，发现静态基准与交互性能相关性弱，需开发更贴合用户偏好的评估。
摘要：
随着AI聊天机器人变得无处不在，语音交互为快速、高带宽的语义和社会信号通信提供了一种引人注目的方式。这推动了大型音频模型（LAMs）的研究，以支持语音原生体验。然而，将LAM开发与用户目标对齐需要清晰理解用户需求和偏好，以建立可靠的进展指标。本研究通过引入一种交互式方法来评估LAMs，并从484名参与者处收集了7,500次LAM交互，以应对这些挑战。通过对用户查询的主题建模，我们识别了音频接口的主要用例。然后，我们分析用户偏好排名和定性反馈，以确定哪些模型最符合用户需求。最后，我们评估静态基准如何预测交互性能——我们的分析显示，没有任何单一基准与交互结果强相关（所有基准的τ≤0.33）。虽然结合多个粗粒度特征能产生适度的预测能力（R²=0.30），但在口语问答和年龄预测的二十个数据集中，只有两个显示出显著正相关。这表明明确需要开发与用户偏好更相关的LAM评估方法。
英文摘要：
As AI chatbots become ubiquitous, voice interaction presents a compelling way to enable rapid, high-bandwidth communication for both semantic and social signals. This has driven research into Large Audio Models (LAMs) to power voice-native experiences. However, aligning LAM development with user goals requires a clear understanding of user needs and preferences to establish reliable progress metrics. This study addresses these challenges by introducing an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions from 484 participants. Through topic modeling of user queries, we identify primary use cases for audio interfaces. We then analyze user preference rankings and qualitative feedback to determine which models best align with user needs. Finally, we evaluate how static benchmarks predict interactive performance - our analysis reveals no individual benchmark strongly correlates with interactive results ($τ\leq 0.33$ for all benchmarks). While combining multiple coarse-grained features yields modest predictive power ($R^2$=$0.30$), only two out of twenty datasets on spoken question answering and age prediction show significantly positive correlations. This suggests a clear need to develop LAM evaluations that better correlate with user preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
