arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.19893cs.AIcs.HC
收藏

利用基础模型进行零样本物联网感知

Leveraging Foundation Models for Zero-Shot IoT Sensing

Delft University of Technology(代尔夫特理工大学);Google(谷歌公司);University of Pittsburgh(匹兹堡大学)

Dinghao Xue, Xiaoran Fan, Tao Chen, Guohao Lan, Qun Song

AI总结:

该研究提出一种利用基础模型进行零样本物联网感知的方法,通过交叉注意力结合软硬提示并使用数据增强合成未见类数据,有效提升了开放集检测和广义零样本学习的性能。

摘要:

深度学习模型越来越多地部署在边缘物联网设备上。然而,这些模型通常在监督条件下运行,无法识别与训练数据不同的未见类别。为了解决这个问题,零样本学习(ZSL)旨在借助语义信息对未见类别的数据进行分类。在网络规模数据上训练的基础模型(FMs)在自然语言处理和视觉理解方面展现出了令人印象深刻的ZSL能力。然而,利用FMs的通用知识进行基于毫米波、IMU和Wi-Fi等信号的零样本物联网感知尚未得到充分研究。在这项工作中,我们将物联网数据嵌入与FM文本编码器生成的语义嵌入进行对齐,以实现零样本物联网感知。为了利用支配物联网传感器信号生成的物理原理来导出更有效的提示以进行语义嵌入提取,我们提出使用交叉注意力机制来结合在训练数据上自动优化的可学习软提示和编码了物联网感知任务领域知识的辅助硬提示。为了解决由于训练期间缺乏未见类别数据而导致物联网嵌入偏向已见类别的问题,我们提出使用数据增强来合成未见类别的物联网数据,以微调物联网特征提取器和嵌入投影器。我们在多个物联网感知任务上评估了我们的方法。结果表明,与各种基线相比,我们的方法实现了优越的开放集检测和广义零样本学习性能。我们的代码可在 https://github.com/schrodingho/FM_ZSL_IoT 获取。

英文摘要:

Deep learning models are increasingly deployed on edge Internet of Things (IoT) devices. However, these models typically operate under supervised conditions and fail to recognize unseen classes different from training. To address this, zero-shot learning (ZSL) aims to classify data of unseen classes with the help of semantic information. Foundation models (FMs) trained on web-scale data have shown impressive ZSL capability in natural language processing and visual understanding. However, leveraging FMs' generalized knowledge for zero-shot IoT sensing using signals such as mmWave, IMU, and Wi-Fi has not been fully investigated. In this work, we align the IoT data embeddings with the semantic embeddings generated by an FM's text encoder for zero-shot IoT sensing. To utilize the physics principles governing the generation of IoT sensor signals to derive more effective prompts for semantic embedding extraction, we propose to use cross-attention to combine a learnable soft prompt that is optimized automatically on training data and an auxiliary hard prompt that encodes domain knowledge of the IoT sensing task. To address the problem of IoT embeddings biasing to seen classes due to the lack of unseen class data during training, we propose using data augmentation to synthesize unseen class IoT data for fine-tuning the IoT feature extractor and embedding projector. We evaluate our approach on multiple IoT sensing tasks. Results show that our approach achieves superior open-set detection and generalized zero-shot learning performance compared with various baselines. Our code is available at https://github.com/schrodingho/FM\_ZSL\_IoT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑