arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.06459cs.CVcs.AI
收藏

基础模型知道几何吗?通过冻结特征进行连续物理测量

Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement

Yakov Pyotr Shkolnikov

AI总结:

研究发现视觉-语言模型通过冻结特征可实现连续几何测量,LoRA微调和架构分析揭示了路径训练缺陷及中层注意力头对几何信号的重要性。

摘要:

视觉-语言模型编码了连续几何,而其文本路径无法表达:一个6000参数的线性探针从冻结特征中提取手关节角度,其MAE为6.1度,而最佳文本输出仅能达到20.0度,这存在3.3倍的瓶颈。LoRA微调(r=16,2000张图像)将这一差距缩小到6.5度,提供了路径训练缺陷的证据,而非表示层面的缺陷。训练目标决定了准确性,而非架构:五个涵盖自监督、对比和混合范式的编码器在统计上等效的准确性(R²约0.55,TOST等效于delta=0.03)下,尽管共享的表示相似性仅为CKA=0.41,实现了功能收敛而无表示收敛。自回归生成破坏了几何保真度,但这种损害源于生成过程,而非语言对齐:Qwen2.5-VL的LLM层实际上比其原始视觉编码器提高了探针的准确性。逐层分析揭示了所有架构中存在一个通用中层准确性峰值,第18-22层的注意力头承载了不成比例的几何信号。这些发现使单个冻结主干能够通过轻量级探针作为多任务几何传感器,而无需微调或文本生成。

英文摘要:

Vision-language models encode continuous geometry that their text pathway fails to express: a 6,000-parameter linear probe extracts hand joint angles at 6.1 degrees MAE from frozen features, while the best text output achieves only 20.0 degrees -- a 3.3x bottleneck. LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees, providing evidence for a pathway-training deficit rather than a representational one. Training objective determines accuracy more than architecture: five encoders spanning self-supervised, contrastive, and hybrid paradigms converge to statistically equivalent accuracy (R^2 approximately 0.55, TOST-equivalent at delta=0.03) despite sharing as little as CKA=0.41 representational similarity -- functional convergence without representational convergence. Autoregressive generation damages geometric fidelity, but the damage originates in the generation process, not in language alignment: Qwen2.5-VL's LLM layers actually improve probe accuracy over its raw vision encoder. Layer-wise analysis reveals a universal mid-network accuracy peak across all architectures, with attention heads in layers 18-22 carrying disproportionate geometric signal. These findings enable a single frozen backbone to function as a multi-task geometric sensor through lightweight probes, without fine-tuning or text generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑