触觉的语言:通过双分支学习将振动转化为文本
The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning
AI总结:
本文首次尝试解决触觉描述生成问题,提出ViPAC方法,通过双分支策略和动态融合机制处理触觉数据的周期性和非周期性特性,构建首个触觉-文本配对数据集LMT108-CAP,实验表明其在词汇准确性和语义对齐方面优于基线方法。
摘要:
IEEE P1918.1工作组对振动触觉数据的标准化极大地推动了其在虚拟现实、人机交互和具身人工智能中的应用。尽管有这些努力,振动触觉信号的语义解释和理解仍是一个未解决的挑战。本文首次尝试解决振动触觉描述生成问题,即从振动触觉信号生成自然语言描述。我们提出了振动周期-非周期描述(ViPAC),一种旨在处理振动触觉数据固有特性的方法,包括混合周期-非周期结构和缺乏空间语义。具体而言,ViPAC采用双分支策略来分离周期性和非周期性成分,结合动态融合机制,以自适应整合信号特征。此外,我们引入正交约束和加权正则化,以确保特征互补性和融合一致性。此外,我们构建了LMT108-CAP,首个触觉-文本配对数据集,利用GPT-4o从流行的LMT-108数据集生成每个表面图像的五个受约束描述。实验表明,ViPAC在词汇准确性和语义对齐方面显著优于从音频和图像描述中改编的基线方法。
英文摘要:
The standardization of vibrotactile data by IEEE P1918.1 workgroup has greatly advanced its applications in virtual reality, human-computer interaction and embodied artificial intelligence. Despite these efforts, the semantic interpretation and understanding of vibrotactile signals remain an unresolved challenge. In this paper, we make the first attempt to address vibrotactile captioning, {\it i.e.}, generating natural language descriptions from vibrotactile signals. We propose Vibrotactile Periodic-Aperiodic Captioning (ViPAC), a method designed to handle the intrinsic properties of vibrotactile data, including hybrid periodic-aperiodic structures and the lack of spatial semantics. Specifically, ViPAC employs a dual-branch strategy to disentangle periodic and aperiodic components, combined with a dynamic fusion mechanism that adaptively integrates signal features. It also introduces an orthogonality constraint and weighting regularization to ensure feature complementarity and fusion consistency. Additionally, we construct LMT108-CAP, the first vibrotactile-text paired dataset, using GPT-4o to generate five constrained captions per surface image from the popular LMT-108 dataset. Experiments show that ViPAC significantly outperforms the baseline methods adapted from audio and image captioning, achieving superior lexical fidelity and semantic alignment.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。