看见声音，听见视觉：揭示AI模型在声音定位中的模态偏差与冲突
Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
AI总结：
本研究系统考察AI声音定位中的模态偏差，发现人类优于AI，并提出受神经科学启发的EchoPin模型，其超越现有基准并呈现类似人类的水平定位偏差。
摘要：
想象一下，你听到狗叫声并转向声音来源，却只看到一辆停着的汽车，而真正的狗正安静地坐在别处。这种感官冲突考验着感知能力，而人类能够可靠地解决这些冲突，其方式是优先处理声音而非误导性的视觉信息。尽管结合视觉和听觉的多模态AI取得了进展，但人们对这些系统如何处理跨模态冲突，或者它们是否偏向某一模态，仍知之甚少。在本研究中，我们系统地考察了AI声音定位中的模态偏差和冲突解决能力。我们评估了领先的多模态模型，并在六种视听条件下（包括一致、冲突和缺失线索）的心理物理学实验中，将它们与人类表现进行基准比较。人类始终优于AI，表现出对冲突或缺失视觉信息的卓越韧性，其方式是依赖听觉信息。相比之下，AI模型往往默认使用视觉输入，导致性能下降到接近随机水平。为解决这一问题，我们提出了一种受神经科学启发的模型EchoPin，该模型使用通过3D模拟生成的立体声图像数据集。即使训练数据有限，EchoPin也超越了现有基准。值得注意的是，它还呈现出类似人类的水平定位偏差，即偏好左右精度——这可能归因于反映人类耳朵布局的立体声结构。这些发现强调了感官输入质量和系统架构如何塑造多模态表征的准确性。
英文摘要：
Imagine hearing a dog bark and turning toward the sound only to see a parked car, while the real, silent dog sits elsewhere. Such sensory conflicts test perception, yet humans reliably resolve them by prioritizing sound over misleading visuals. Despite advances in multimodal AI integrating vision and audio, little is known about how these systems handle cross-modal conflicts or whether they favor one modality. In this study, we systematically examine modality bias and conflict resolution in AI sound localization. We assess leading multimodal models and benchmark them against human performance in psychophysics experiments across six audiovisual conditions, including congruent, conflicting, and absent cues. Humans consistently outperform AI, demonstrating superior resilience to conflicting or missing visuals by relying on auditory information. In contrast, AI models often default to visual input, degrading performance to near chance levels. To address this, we propose a neuroscience-inspired model, EchoPin, which uses a stereo audio-image dataset generated via 3D simulations. Even with limited training data, EchoPin surpasses existing benchmarks. Notably, it also mirrors human-like horizontal localization bias favoring left-right precision-likely due to the stereo audio structure reflecting human ear placement. These findings underscore how sensory input quality and system architecture shape multimodal representation accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
