高维跨模态对齐冻结语言和图像模型以实现高效的图像描述生成
Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning
AI总结：
HDFLIM通过冻结语言和图像模型的高维计算实现高效的图像描述生成，无需参数调优。
摘要：
大型单模基础模型在视觉和语言领域编码了丰富的语义结构，但通常需要计算密集的多模态微调来对齐它们。此类方法依赖于大规模参数更新，资源消耗大，且可能扰动预训练表示。然而，越来越多的证据表明，独立训练的基础模型可能已经表现出潜在的语义兼容性，反映了它们所建模的数据中的共享结构。这提出了一个根本性的问题：能否在不修改模型本身的情况下实现跨模态对齐？在此，我们介绍了HDFLIM（使用冻结语言和图像模型的高维计算），一个框架，它在保持预训练视觉和语言模型完全冻结的情况下建立跨模态映射。HDFLIM将单模嵌入投影到共享的高维空间，并利用轻量级的符号操作——绑定、捆绑和基于相似性的检索，通过单次数据遍历构建关联的跨模态表示。描述生成源于高维记忆检索，而不是迭代梯度优化。我们展示了HDFLIM在性能上与端到端视觉-语言训练方法相当，并且生成的描述比零样本基线更具语义基础性。通过将对齐与参数调优解耦，我们的结果表明，通过在各自嵌入的高维编码上进行符号操作，可以在基础模型之间实现语义映射。更广泛地说，这项工作指出了一个替代的基础模型对齐范式，在这种范式中，冻结模型通过结构化的表示映射进行整合，而不是通过大规模重新训练。我们的实现代码库可在https://github.com/Abhishek-Dalvi410/HDFLIM上找到。
英文摘要：
Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Such approaches depend on large-scale parameter updates, are resource intensive, and can perturb pretrained representations. Emerging evidence suggests, however, that independently trained foundation models may already exhibit latent semantic compatibility, reflecting shared structures in the data they model. This raises a fundamental question: can cross-modal alignment be achieved without modifying the models themselves? Here we introduce HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen. HDFLIM projects unimodal embeddings into a shared hyperdimensional space and leverages lightweight symbolic operations -- binding, bundling, and similarity-based retrieval to construct associative cross-modal representations in a single pass over the data. Caption generation emerges from high-dimensional memory retrieval rather than iterative gradient-based optimization. We show that HDFLIM achieves performance comparable to end-to-end vision-language training methods and produces captions that are more semantically grounded than zero-shot baselines. By decoupling alignment from parameter tuning, our results suggest that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings. More broadly, this work points toward an alternative paradigm for foundation model alignment in which frozen models are integrated through structured representational mappings rather than through large-scale retraining. The codebase for our implementation can be found at https://github.com/Abhishek-Dalvi410/HDFLIM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
