醒来触摸！多模态大语言模型中的掩码隔离触觉对齐学习
Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs
AI总结：
提出Splash框架，通过参数空间划分（休眠/关键子空间）实现非破坏性触觉模态扩展，在不增加推理开销下保持视觉语言能力，在触觉基准上达到SOTA。
摘要：
触觉提供了感知内在材料属性（如摩擦和顺应性）所需的物理基础，而仅靠视觉通常无法解决这些问题。然而，最近为多模态大语言模型配备这种触觉感知的努力暴露了一种零和权衡：紧凑模型的有限参数预算迫使在新感官模态的获取与已建立的视觉语言推理之间做出选择。我们提出了Splash，一种用于MLLMs的掩码隔离触觉对齐学习框架。Splash量化每个预训练参数的重要性，并将参数空间划分为休眠和关键子空间。冻结的关键子空间作为稳定锚点以保护通用视觉知识，同时Splash更新隔离的休眠子空间以将触觉对齐内化到LLMs中。这种选择性、非破坏性的扩展有效防止了灾难性遗忘，并确保了非破坏性的模态扩展。大量实验表明，Splash在不增加LLM部分推理开销的情况下有效实现了触觉推理，在包括SSVTP、TVL和TacQuad在内的视觉-触觉基准上展示了最先进的性能，同时保持了其原有的通用能力。
英文摘要：
Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
