arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.22221cs.CVcs.LGcs.SD
收藏

视觉能从音频中学到多少物理知识?探测与蒸馏阻尼结构

How Much Physics Can Vision Learn from Audio? Probing and Distilling Damping Structure

Amazon Web Services(亚马逊网络服务)

Zainab Afolabi, Vadim Selyutin, Zhao Gao, Babs Khalidson, Mohamed Ghoneim

AI总结:

该研究探究视觉从音频学习物理阻尼结构的能力,通过蒸馏音频知识到CLIP骨干网络,提升视觉对材料阻尼的排序性能,揭示可见动态的作用。

摘要:

物理理解对决策至关重要,但视觉模型难以推断阻尼等决定材料行为的属性。音频直接编码这类属性,因为物体受撞击后的声学响应可通过可测量的衰减揭示其阻尼特性。我们探测冻结编码器的品质因数Q(无量纲量,衡量物体振动衰减前完成的振荡次数),并采用斯皮尔曼秩相关系数ρ评估各嵌入对撞击按Q排序的效果,其中ρ=1为完美排序,ρ=0为无关联。音频嵌入达到ρ=0.73,在训练未包含的材料上仍保留大部分优势;而CLIP编码的撞击瞬间单帧视频仅支持ρ=0.28。明确信号来源后,我们探究视觉能弥补多少差距。我们通过训练投影头将冻结音频教师模型蒸馏到冻结CLIP骨干网络,发现这可恢复音频优势的一部分。由于该任务的池化相关性实际受类别主导,我们还采用留一材料协议评估,即投影头和探测均排除某一材料。四帧输入的蒸馏使该协议下的相关性从0.234提升至0.338(较匹配四帧基线提升0.053,17种材料中有14种实现),这是类别级策略无法产生的增益。将视觉输入扩展至撞击前后的四帧,可在无声学成本的情况下提供一半的改进,表明单帧缺失的部分是可见动态。

英文摘要:

Physical understanding is crucial for decision making, yet vision models struggle to infer properties such as damping that govern how materials behave. Audio encodes such properties directly, because an object's acoustic response to impact reveals its damping through measurable decay. We probe frozen encoders for the quality factor $Q$, a dimensionless measure of how many oscillations an object completes before its ring dies away, and report how well each embedding orders impacts by $Q$ using the Spearman rank correlation $ρ$, for which $ρ=1$ is a perfect ordering and $ρ=0$ an unrelated one. Audio embeddings reach $ρ=0.73$ and retain most of that advantage on materials withheld from training, while a single video frame taken at the moment of impact, encoded by CLIP, supports only $ρ=0.28$. Having established where the signal lives, we ask how much of that gap vision can recover. We distill a frozen audio teacher into a frozen CLIP backbone through a trained projection head and find that this recovers a useful fraction of the audio advantage. Because pooled correlations on this task are dominated in practice by category identity, we also evaluate under a leave-one-material-out protocol in which the held-out material is withheld from the projection head as well as the probe. Distillation with a four-frame input improves the correlation there from 0.234 to 0.338 (+0.053 over the matched four-frame baseline, 14 of 17 materials), a gain that no category-level strategy can produce. Widening the visual input to four frames spanning the impact supplies half of that improvement at no acoustic cost, which indicates that part of what a single frame misses is visible dynamics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑