arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.07408cs.AI
收藏

基于文本嵌入反转的传感器开放词汇活动识别初步研究

Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion

DFKI(德国人工智能研究中心);RPTU(莱茵兰-普法尔茨州立大学凯撒斯劳滕-兰道分校)

Lala Shakti Swarup Ray, Bo Zhou, Sungho Suh, Paul Lukowicz

AI总结:

针对传统人体活动识别仅能识别训练集内活动的局限,提出OV-HAR框架,通过文本嵌入与反转实现无自回归LLM的开放词汇活动识别,在多模态上验证了强泛化性。

摘要:

传统人体活动识别(HAR)依赖经训练预测离散活动类别的分类器,本质上将识别范围局限于训练集中明确存在的活动。这类分类器在遇到未见过的活动时必然失效,给出的概率为零。我们提出开放词汇HAR(OV-HAR)框架,该框架首先将每项活动转换为自然语言并拆解为一系列基本动作,以此突破上述限制。随后这段描述文本被编码为固定大小的嵌入向量。模型经训练对该嵌入向量进行回归,之后利用预训练的嵌入反转模型将其解码回自然语言。与其他以自回归大语言模型(LLM)为核心的研究不同,OV-HAR无需这类模型的计算开销即可实现开放词汇识别。生成的文本可通过LLM提示工程转换为单个活动类别。我们已在视觉(姿态)、IMU、压力传感器等不同模态上评估了该方法,证明其在未见过的活动和模态上均具备强泛化能力,提供了一种与现有分类器截然不同的范式。

英文摘要:

Conventional human activity recognition (HAR) relies on classifiers trained to predict discrete activity classes, inherently limiting recognition to activities explicitly present in the training set. Such classifiers would invariably fail, putting zero likelihood, when encountering unseen activities. We propose Open Vocabulary HAR (OV-HAR), a framework that overcomes this limitation by first converting each activity into natural language and breaking it into a sequence of elementary motions. This descriptive text is then encoded into a fixed-size embedding. The model is trained to regress this embedding, which is subsequently decoded back into natural language using a pre-trained embedding inversion model. Unlike other works that rely on auto-regressive large language models (LLMs) at their core, OV-HAR achieves open vocabulary recognition without the computational overhead of such models. The generated text can be transformed into a single activity class using LLM prompt engineering. We have evaluated our approach on different modalities, including vision (pose), IMU, and pressure sensors, demonstrating robust generalization across unseen activities and modalities, offering a fundamentally different paradigm from contemporary classifiers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑