在智能环境中通过非侵入式传感器识别日常活动，可支撑多种医疗健康应用。监测受试者在家中的活动执行方式及其随时间的变化，能够发现健康问题的早期症状，例如认知衰退。该领域的大多数方法采用深度学习模型，这类模型通常被视为将传感器数据映射到活动的黑盒。然而，临床医生等非专业用户需要信任并理解这些模型的输出。因此，面向人类活动识别的可解释人工智能（eXplainable AI, XAI）方法应运而生，旨在从这些模型生成直观的自然语言解释。不同的XAI方法会生成不同的解释，其有效性通常通过用户调研来评估，而这类调研往往在成本和公平性方面存在挑战。本文提出一种使用大语言模型（Large Language Models, LLMs）的自动评估方法，用于在候选方法池中筛选出最适合非专业用户的XAI方法。我们的初步结果表明，LLM评估结果与用户调研结果具有一致性。
英文摘要：
Recognizing daily activities with unobtrusive sensors in smart environments enables various healthcare applications. Monitoring how subjects perform activities at home and their changes over time can reveal early symptoms of health issues, such as cognitive decline. Most approaches in this field use deep learning models, which are often seen as black boxes mapping sensor data to activities. However, non-expert users like clinicians need to trust and understand these models' outputs. Thus, eXplainable AI (XAI) methods for Human Activity Recognition have emerged to provide intuitive natural language explanations from these models. Different XAI methods generate different explanations, and their effectiveness is typically evaluated through user surveys, that are often challenging in terms of costs and fairness. This paper proposes an automatic evaluation method using Large Language Models (LLMs) to identify, in a pool of candidates, the best XAI approach for non-expert users. Our preliminary results suggest that LLM evaluation aligns with user surveys.
SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes
Midea Group（美的集团）；Beijing University of Posts and Telecommunications（北京邮电大学）；Donghua University（东华大学）；The University of Sydney（悉尼大学）；Peking University（北京大学）
SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Jilin University（吉林大学）；The Chinese University of Hong Kong（香港中文大学）；Pengcheng Laboratory（鹏城实验室）
Identifying and Addressing User-level Security Concerns in Smart Homes Using "Smaller" LLMs
Department of CSE Bangladesh University of Engineering（计算机科学与工程系 bangladesh 工程大学）；Concordia Institute for Information Systems Engineering Concordia University Montreal, Canada（信息系统工程研究所 博士大学 蒙特利尔 加拿大）
SmartHomeSecure: Automated Detection and Repair of Smart Home Configuration Errors Using Large Language Models
Myers-Lawson School of Construction, Virginia Polytechnic Institute and State University（Myers-Lawson工程学院，弗吉尼亚理工学院和州立大学）；School of Technology, Eastern Illinois University（技术学院，东伊利诺伊大学）
Synthetic User Behavior Sequence Generation with Large Language Models for Smart Homes
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；PengCheng Laboratory（鹏城实验室）；The Chinese University of Hong Kong（香港中文大学）；University of Warwick（华威大学）
Designing smart home services is a complex task when multiple services with a large number of sensors and actuators are deployed simultaneously. It may rely on knowledge-based or data-driven approaches. The former can use rule-based methods to design services statically, and the latter can use learning methods to discover inhabitants' preferences dynamically. However, neither of these approaches is entirely satisfactory because rules cannot cover all possible situations that may change, and learning methods may make decisions that are sometimes incomprehensible to the inhabitant. In this paper…
Explaining AI Decisions: Towards Achieving Human-Centered Explainability in Smart Home Environments
Fraunhofer Institute for Applied Information Technology FIT（弗劳恩霍夫应用信息技术研究所）；University of Siegen（锡根大学）；Bonn-Rhein-Sieg University of Applied Sciences（波恩-莱茵-锡格应用科学大学）
Recent advances in Internet of Things (IoT) technologies and the reduction in the cost of sensors have encouraged the development of smart environments, such as smart homes. Smart homes can offer home assistance services to improve the quality of life, autonomy and health of their residents, especially for the elderly and dependent. To provide such services, a smart home must be able to understand the daily activities of its residents. Techniques for recognizing human activity in smart homes are advancing daily. But new challenges are emerging every day. In this paper, we present recent algori…
Current home appliances are capable to execute a limited number of voice commands such as turning devices on or off, adjusting music volume or light conditions. Recent progress in machine reasoning gives an opportunity to develop new types of conversational user interfaces for home appliances. In this paper, we apply state-of-the-art visual reasoning model and demonstrate that it is feasible to ask a smart fridge about its contents and various properties of the food with close-to-natural conversation experience. Our visual reasoning model answers user questions about existence, count, category…
Voice control in the smart home is commonplace, enabling the convenient control of smart home Internet of Things hubs, gateways and devices, along with information seeking dialogues. Cloud-based voice assistants are used to facilitate the interaction, yet privacy concerns surround the cloud analysis of data. To what extent can voice control be performed using purely local computation, to ensure user data remains private? In this paper we present a taxonomy of the voice control technologies present in commercial smart home systems. We first review literature on the topic, and summarise relevant…
Layout Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST)
School of Interactive Computing, Georgia Institute of Technology（佐治亚理工学院交互计算学院）；School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）
SPOK is an End-User Development Environment that permits people to monitor, control, and configure smart home services and devices. SPOK has been deployed for more than 4 months in the homes of 5 project team members for testing and refinement, prior to longitudinal experiments in the homes of families not involved in the project. This article reports on the lessons learned in this initial deployment.
Generally, Human Activity Recognition (HAR) consists of monitoring and analyzing the behavior of one or more persons in order to deduce their activity. In a smart home context, the HAR consists of monitoring daily activities of the residents. Thanks to this monitoring, a smart home can offer home assistance services to improve quality of life, autonomy and health of their residents, especially for elderly and dependent people.
Smart home Internet of Things (IoT) devices are rapidly increasing in popularity, with more households including Internet-connected devices that continuously monitor user activities. In this study, we conduct eleven semi-structured interviews with smart home owners, investigating their reasons for purchasing IoT devices, perceptions of smart home privacy risks, and actions taken to protect their privacy from those external to the home who create, manage, track, or regulate IoT devices and/or their data. We note several recurring themes. First, users' desires for convenience and connectedness d…
The home is often the most private space in people's lives, and not one in which they expect to be surveilled. However, today's market for smart home devices has quickly evolved to include products that monitor, automate, and present themselves as human. After documenting some of the more unusual emergent problems with contemporary devices, this body of work seeks to develop a design philosophy for intelligent agents in the smart home that can act as an alternative to the ways that these devices are currently built. This is then applied to the design of privacy empowering technologies, represe…
Recent changes to data protection regulation, particularly in Europe, are changing the design landscape for smart devices, requiring new design techniques to ensure that devices are able to adequately protect users' data. A particularly interesting space in which to explore and address these challenges is the smart home, which presents a multitude of difficult social and technical problems in an intimate and highly private context. This position paper outlines the motivation and research approach of a new project aiming to inform the future of data protection by design and by default in smart …
The automated home referred to as Smart Home is expected to offer fully customized services to its residents, reducing the amount of home labor, thus improving human beings' welfare. Service robots and Internet of Things (IoT) play the key roles in the development of Smart Home. The service provision with these two main components in a Smart Home environment requires: 1) learning and reasoning algorithms and 2) the integration of robot and IoT systems. Conventional computational intelligence-based learning and reasoning algorithms do not successfully manage dynamic changes in the Smart Home da…
In recent years the concept of the Internet of Things (IoT) has evolved to connect commercial gadgets together with the medical field to facilitate an unprecedented range of accessibility. The development of medical devices connected to internet of things has been praised for the potential of alleviating the strain on the modern healthcare system by giving users the opportunity to reside in the home during treatment or recovery. With the IoT becoming more prevalent and available at a commercial level, there exists room for integration into emerging, intelligent environments such as smart homes…
With the growing advances in the Internet of Things (IoT) technology, IoT device management platforms are becoming increasingly important. We conducted a web-based survey and usability study with 43 participants who use IoT devices frequently to: 1) examine their smart home IoT usage patterns and privacy preferences, and 2) evaluate a web-based prototype for smart home IoT device management. We found that participants perceived privacy as more important than the convenience afforded by the IoT devices. Based on their average scores of privacy vs. convenience importance, participants with low p…
Device use in smart homes is becoming increasingly communal, requiring cohabitants to navigate a complex social and technological context. In this paper, we report findings from an exploratory survey grounded in our prior work on communal technology use in the home [4]. The findings highlight the importance of considering qualities of social relationships and technology in understanding expectations and intentions of communal technology use. We propose a design perspective of social expectations, and we suggest existing designs can be expanded using already available information such as locati…
The increasingly widespread use of 'smart' devices has raised multifarious ethical concerns regarding their use in domestic spaces. Previous work examining such ethical dimensions has typically either involved empirical studies of concerns raised by specific devices and use contexts, or alternatively expounded on abstract concepts like autonomy, privacy or trust in relation to 'smart homes' in general. This paper attempts to bridge these approaches by asking what features of smart devices users consider as rendering them 'smart' and how these relate to ethical concerns. Through a multimethod i…
The proliferation of Internet of Things (IoT) devices for consumer "smart" homes raises concerns about user privacy. We present a survey method based on the Contextual Integrity (CI) privacy framework that can quickly and efficiently discover privacy norms at scale. We apply the method to discover privacy norms in the smart home context, surveying 1,731 American adults on Amazon Mechanical Turk. For $2,800 and in less than six hours, we measured the acceptability of 3,840 information flows representing a combinatorial space of smart home devices sending consumer information to first and third-…
Recently, smart home systems based on brain-computer interface (BCI) has attracted a wide range of interests in both industry and academia. However, the current BCI system has several shortcomings as it produces a comparatively lower accuracy for real-time implementations as well as the intuitive paradigm for the users cannot be well established here. Therefore, in this study, we proposed a highly intuitive BCI paradigm that combines event-related potential (ERP) with the speech-imagery task for the individual target objects. The decoding accuracy of the proposed paradigm was 88.1% (plus or mi…
The advent of IoT has enabled the design of connected and integrated smart health monitoring systems. These smart health monitoring systems could be realized in a smart home context to render long-term care to the elderly population. In this paper, we present the design of a wearable health monitoring system suitable for older adults in a smart home context. The proposed system offers solutions to monitor the stress, blood pressure, and location of an individual within a smart home environment. The stress detection model proposed in this work uses Electrodermal Activity (EDA), Photoplethysmogr…
This chapter introduces the Accountability Principle and its role in data protection governance. We focus on what accountability means in the context of cybersecurity management in smart homes, considering the EU General Data Protection Law requirements to secure personal data. This discussion sits against the backdrop of two key new developments in data protection law. Firstly, the law is moving into the home, due to narrowing of the so called household exemption. Concurrently, household occupants may now have legal responsibilities to comply with the GDPR, as they find themselves jointly res…
TRACE: Temporal Reasoning over Context and Evidence for Activity Recognition in Smart Homes
School of Interactive Computing, Georgia Institute of Technology（佐治亚理工学院互动计算学院）；Georgia Institute of Technology（佐治亚理工学院）；Berry College（贝瑞学院）；Northeastern University（东北大学）；Universidad Carlos III de Madrid（马德里卡洛斯三世大学）
Investigating Bystander Privacy in Chinese Smart Home Apps
King’s College London（伦敦国王学院）；University of St Andrews（圣安德鲁斯大学）；University of Warwick（华威大学）；INGENIO (CSIC-Universitat Politècnica de València)（INGENIO（西班牙国家研究委员会-巴伦西亚理工大学））
This study discloses the development of a solution for realizing a smart home in the post COVID-19 era using the Internet of Things domain knowledge. COVID-19 outbreak has been catastrophic and impacted everyone's lives due to its rapid transmission from one body to another. This study aims to reduce virus transmission by eliminating the need to touch any common-point surface in a home, such as switches, doorknobs, and remotes. We provide a generic solution by coupling things with the internet to control them remotely. The project aims to showcase a working solution to controlling devices like…
This paper gives an overview of the assessment and evaluation methods which have been used to determine the quality of the INSPIRE smart home system. The system allows different home appliances to be controlled via speech, and consists of speech and speaker recognition, speech understanding, dialogue management, and speech output components. The performance of these components is first assessed individually, and then the entire system is evaluated in an interaction experiment with test users. Initial results of the assessment and evaluation are given, in particular with respect to the transmis…
