SoundLoc3D：基于多模态RGB-D声学相机的不可见3D声源定位与分类
SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera
AI总结：
SoundLoc3D利用RGB-D相机与四通道麦克风阵列构建多模态声学相机，将不可见3D声源定位与分类建模为集合预测问题，通过视听跨模态线索与物理表面约束实现高精度定位，并在模拟数据上验证了其有效性和鲁棒性。
摘要：
精确地定位3D声源并估计其语义标签——其中声源可能不可见，但假设位于场景中物体的物理表面上——具有许多实际应用，包括检测气体泄漏和机械故障。这种场景下的视听弱相关性为推导创新方法以回答是否或如何利用跨模态信息解决该任务提出了新的挑战。为此，我们提出使用由针孔RGB-D相机和共面四通道麦克风阵列（Mic-Array）组成的声学相机装置。通过使用该装置从多视角记录视听信号，我们可以利用跨模态线索来估计声源的3D位置。具体而言，我们的框架SoundLoc3D将该任务视为集合预测问题，集合中的每个元素对应一个潜在的声源。考虑到视听弱相关性，集合表示最初从单视角麦克风阵列信号中学习，然后通过主动融合从多视角RGB-D图像中揭示的物理表面线索进行细化。我们在大规模模拟数据集上证明了SoundLoc3D的效率和优越性，并进一步展示了其对RGB-D测量不准确性和环境噪声干扰的鲁棒性。
英文摘要：
Accurately localizing 3D sound sources and estimating their semantic labels -- where the sources may not be visible, but are assumed to lie on the physical surface of objects in the scene -- have many real applications, including detecting gas leak and machinery malfunction. The audio-visual weak-correlation in such setting poses new challenges in deriving innovative methods to answer if or how we can use cross-modal information to solve the task. Towards this end, we propose to use an acoustic-camera rig consisting of a pinhole RGB-D camera and a coplanar four-channel microphone array~(Mic-Array). By using this rig to record audio-visual signals from multiviews, we can use the cross-modal cues to estimate the sound sources 3D locations. Specifically, our framework SoundLoc3D treats the task as a set prediction problem, each element in the set corresponds to a potential sound source. Given the audio-visual weak-correlation, the set representation is initially learned from a single view microphone array signal, and then refined by actively incorporating physical surface cues revealed from multiview RGB-D images. We demonstrate the efficiency and superiority of SoundLoc3D on large-scale simulated dataset, and further show its robustness to RGB-D measurement inaccuracy and ambient noise interference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
