IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor data
MARS Lab, School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（MARS实验室，机械与航空航天工程学院，南洋理工大学，新加坡）；School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（机械与航空航天工程学院，南洋理工大学，新加坡）
Large Language Models excel in textual tasks but often struggle with physical-world reasoning tasks. Inspired by human cognition, where perception is fundamental to reasoning, we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) data and pertinent knowledge. In this work, we systematically study LLMs' capability to address IoT-sensory tasks by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps: preprocessing IoT data into suitable formats, expanding LLMs knowledge via IoT-oriented retrieval-augmented generation and activating LLMs commonsense knowledge through chain-of-thought prompting. We design a benchmark comprising five real-world tasks with varying data types and reasoning complexities to evaluate the performance of IoT-LLM. Experimental results reveal that IoT-LLM significantly improves the performance of IoT-sensory task reasoning of LLMs, with models like GPT-4o-mini showing a 49.4% average improvement over previous methods.
Indraprastha Institute of Information Technology（因德拉普拉斯塔信息技术学院）；Inria（法国国家信息与自动化研究所）
Several domains have adopted the increasing use of IoT-based devices to collect sensor data for generating abstractions and perceptions of the real world. This sensor data is multi-modal and heterogeneous in nature. This heterogeneity induces interoperability issues while developing cross-domain applications, thereby restricting the possibility of reusing sensor data to develop new applications. As a solution to this, semantic approaches have been proposed in the literature to tackle problems related to interoperability of sensor data. Several ontologies have been proposed to handle different …
Internet-of-Things (IoT) devices are often used to transmit physical sensor data over digital wireless channels. Traditional Physical Layer Security (PLS)-based cryptography approaches rely on accurate channel estimation and information exchange for key generation, which irrevocably ties key quality with digital channel estimation quality. Recently, we proposed a new concept called Graph Layer Security (GLS), where digital keys are derived from physical sensor readings. The sensor readings between legitimate users are correlated through a common background infrastructure environment (e.g., a c…
University of Oulu（奥卢大学）；Linköping University（林雪平大学）
We consider an IoT sensing network with multiple users, multiple energy harvesting sensors, and a wireless edge node acting as a gateway between the users and sensors. The users request for updates about the value of physical processes, each of which is measured by one sensor. The edge node has a cache storage that stores the most recently received measurements from each sensor. Upon receiving a request, the edge node can either command the corresponding sensor to send a status update, or use the data in the cache. We aim to find the best action of the edge node to minimize the average long-te…
University College Dublin（都柏林大学学院）；New Jersey Institute of Technology（新泽西理工学院）
The Internet of Things (IoT) is a paradigm characterized by a network of embedded sensors and services. These sensors are incorporated to collect various information, track physical conditions, e.g., waste bins' status, and exchange data with different centralized platforms. The need for such sensors is increasing; however, proliferation of technologies comes with various challenges. For example, how can IoT and its associated data be used to enhance waste management? In smart cities, an efficient waste management system is crucial. Artificial Intelligence (AI) and IoT-enabled approaches can e…
Enhancing Adversarial Robustness of IoT Intrusion Detection via SHAP-Based Attribution Fingerprinting
School of Information Technology, York University（约克大学信息科技学院）；School of Computer Science, University of Guelph（圭尔夫大学计算机科学学院）；National Research Council of Canada（加拿大国家研究委员会）
This article explores the concepts of online protocol synthesis using Reinforcement Learning (RL). The study is performed in the context of sensor and IoT networks with ultra low complexity wireless transceivers. The paper introduces the use of RL and Multi Armed Bandit (MAB), a specific type of RL, for Medium Access Control (MAC) under different network and traffic conditions. It then introduces a novel learning based protocol synthesis framework that addresses specific difficulties and limitations in medium access for both random access and time slotted networks. The mechanism does not rely …
Currently, most social robots interact with their surroundings and humans through sensors that are integral parts of the robots, which limits the usability of the sensors, human-robot interaction, and interchangeability. A wearable sensor garment that fits many robots is needed in many applications. This article presents an affordable wearable sensor vest, and an open-source software architecture with the Internet of Things (IoT) for social humanoid robots. The vest consists of touch, temperature, gesture, distance, vision sensors, and a wireless communication module. The IoT feature allows th…
Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodology
Aqaba University of Technology（阿卡巴理工大学）；Concordia University（康考迪亚大学）；Khalifa University（哈利法大学）；The University of Jordan（约旦大学）；Applied Science Private University（应用科学私立大学）；King Fahd University of Petroleum & Minerals（法赫德国王石油与矿业大学）
AI Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；Tantiv4（Tantiv4公司）
The state identification problem seeks to identify power usage patterns of any system, like buildings or factories, of interest. In this challenge paper, we make power usage dataset available from 8 institutions in manufacturing, education and medical institutions from the US and India, and an initial un-supervised machine learning based solution as a baseline for the community to accelerate research in this area.
Recent advances in Internet of Things (IoT) technologies and the reduction in the cost of sensors have encouraged the development of smart environments, such as smart homes. Smart homes can offer home assistance services to improve the quality of life, autonomy and health of their residents, especially for the elderly and dependent. To provide such services, a smart home must be able to understand the daily activities of its residents. Techniques for recognizing human activity in smart homes are advancing daily. But new challenges are emerging every day. In this paper, we present recent algori…
Resilient Machine-learning Institute (ReMI)（弹性机器学习研究所）
In wireless Internet of things (IoT), the sensors usually have limited bandwidth and power resources. Therefore, in a distributed setup, each sensor should compress and quantize the sensed observations before transmitting them to a fusion center (FC) where a global decision is inferred. Most of the existing compression techniques and entropy quantizers consider only the reconstruction fidelity as a metric, which means they decouple the compression from the sensing goal. In this work, we argue that data compression mechanisms and entropy quantizers should be co-designed with the sensing goal, s…
Detecting patterns in real time streaming data has been an interesting and challenging data analytics problem. With the proliferation of a variety of sensor devices, real-time analytics of data from the Internet of Things (IoT) to learn regular and irregular patterns has become an important machine learning problem to enable predictive analytics for automated notification and decision support. In this work, we address the problem of learning an irregular human activity pattern, fall, from streaming IoT data from wearable sensors. We present a deep neural network model for detecting fall based …
