基于声音的密集2D-3D室内预测：通过对齐的跨模态蒸馏
Dense 2D-3D Indoor Prediction with Sound via Aligned Cross-Modal Distillation
AI总结：
本文提出SAM蒸馏框架，通过匹配实现视觉到音频的空间对齐，首次在DAPS基准上利用声音完成2D和3D密集室内预测，并在深度估计、语义分割和3D重建任务中取得最优性能。
摘要：
声音在我们日常生活中能够为空间推理传递重要信息。为了赋予深度网络这种能力，我们通过跨模态知识蒸馏来解决在2D和3D中利用声音进行密集室内预测的挑战。在这项工作中，我们提出了一种名为“通过匹配实现空间对齐”（Spatial Alignment via Matching，SAM）的蒸馏框架，该框架在视觉到音频的知识迁移中引出两种模态之间的局部对应关系。SAM将音频特征与视觉上一致的可学习空间嵌入相整合，以解决学生模型多个层中的不一致性问题。我们的方法不依赖于特定的输入表示，从而允许输入形状或维度具有灵活性，而不会造成性能下降。借助一个新整理的名为“密集听觉环境预测”（Dense Auditory Prediction of Surroundings，DAPS）的基准，我们首次利用音频观测来处理全向周围环境在2D和3D中的密集室内预测。具体而言，对于基于音频的深度估计、语义分割以及具有挑战性的3D场景重建，所提出的蒸馏框架在各种指标和骨干架构上均持续取得最先进的性能。
英文摘要：
Sound can convey significant information for spatial reasoning in our daily lives. To endow deep networks with such ability, we address the challenge of dense indoor prediction with sound in both 2D and 3D via cross-modal knowledge distillation. In this work, we propose a Spatial Alignment via Matching (SAM) distillation framework that elicits local correspondence between the two modalities in vision-to-audio knowledge transfer. SAM integrates audio features with visually coherent learnable spatial embeddings to resolve inconsistencies in multiple layers of a student model. Our approach does not rely on a specific input representation, allowing for flexibility in the input shapes or dimensions without performance degradation. With a newly curated benchmark named Dense Auditory Prediction of Surroundings (DAPS), we are the first to tackle dense indoor prediction of omnidirectional surroundings in both 2D and 3D with audio observations. Specifically, for audio-based depth estimation, semantic segmentation, and challenging 3D scene reconstruction, the proposed distillation framework consistently achieves state-of-the-art performance across various metrics and backbone architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
