利用视觉与参与者关联输入多模态融合的自主声景增强
Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs
AI总结：
本文改进基于注意力的深度神经网络，融合参与者、视觉与声学特征，证明多模态情境信息可显著提升声景增强预测性能并增强可解释性。
摘要：
自主声景增强系统通常使用经过训练的模型来选择最佳掩蔽声，以实现期望的感知变化。尽管声学信息对此类系统至关重要，但包括参与者人口统计特征和视觉环境在内的情境信息也会影响声学感知。因此，我们对现有基于注意力的深度神经网络提出模块化修改，以实现参与者关联特征、视觉特征和声学特征的早期、中层和晚期特征融合。使用 ARAUS 数据集对模块配置及相应融合方法进行的消融研究表明，情境特征在归一化 ISO 愉悦度上以具有统计显著性的方式提升了模型性能：表现最佳的全模态模型均方误差为 0.1194±0.0012，而纯音频模型为 0.1217±0.0009。因此，声景增强系统可以利用多模态输入来提升性能。我们还使用训练后的模型研究了各个参与者关联因素的影响，以说明模型可解释性的改善。
英文摘要：
Autonomous soundscape augmentation systems typically use trained models to pick optimal maskers to effect a desired perceptual change. While acoustic information is paramount to such systems, contextual information, including participant demographics and the visual environment, also influences acoustic perception. Hence, we propose modular modifications to an existing attention-based deep neural network, to allow early, mid-level, and late feature fusion of participant-linked, visual, and acoustic features. Ablation studies on module configurations and corresponding fusion methods using the ARAUS dataset show that contextual features improve the model performance in a statistically significant manner on the normalized ISO Pleasantness, to a mean squared error of $0.1194\pm0.0012$ for the best-performing all-modality model, against $0.1217\pm0.0009$ for the audio-only model. Soundscape augmentation systems can thereby leverage multimodal inputs for improved performance. We also investigate the impact of individual participant-linked factors using trained models to illustrate improvements in model explainability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
