This demo presents a novel end-to-end framework that combines on-device large language models (LLMs) with smartphone sensing technologies to achieve context-aware and personalized services. The framework addresses critical limitations of current personalization solutions via cloud LLMs, such as privacy concerns, latency and cost, and limited personal information. To achieve this, we innovatively proposed deploying LLMs on smartphones with multimodal sensor data through context-aware sensing and customized prompt engineering, ensuring privacy and enhancing personalization performance. A case study involving a university student demonstrated the capability of the framework to provide tailored recommendations. In addition, we show that the framework achieves the best trade-off in privacy, performance, latency, cost, battery and energy consumption between on-device and cloud LLMs. To the best of our knowledge, this is the first framework to provide on-device LLMs personalization with smartphone sensing. Future work will incorporate more diverse sensor data and involve extensive user studies to enhance personalization. Our proposed framework has the potential to substantially improve user experiences across domains including healthcare, productivity, and entertainment.
PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent
Northwestern Polytechnical University（西北工业大学）；Tsinghua University（清华大学）；Peking University（北京大学）；Beijing Institute of Mathematical Sciences and Applications（北京数学科学与应用研究院）
Over the years, mobile phones have become versatile devices with a multitude of capabilities due to the plethora of embedded sensors that enable them to capture rich data unobtrusively. In a world where people are more conscious regarding their health and well-being, the pervasiveness of smartphones has enabled researchers to build apps that assist people to live healthier lifestyles, and to diagnose and monitor various health conditions. Motivated by the high smartphone coverage among young adults and the unique issues they face, in this review paper, we focus on studies that have used smartp…
Facial expressions have been considered a metric reflecting a person's engagement with a task. While the evolution of expression detection methods is consequential, the foundation remains mostly on image processing techniques that suffer from occlusion, ambient light, and privacy concerns. In this paper, we propose ExpresSense, a lightweight application for standalone smartphones that relies on near-ultrasound acoustic signals for detecting users' facial expressions. ExpresSense has been tested on different users in lab-scaled and large-scale studies for both posed as well as natural expressio…
According to prior work, the type of relationship between the person consuming alcohol and others in the surrounding (friends, family, spouse, etc.), and the number of those people (alone, with one person, with a group, etc.) are related to many aspects of alcohol consumption, such as the drinking amount, location, motives, and mood. Even though the social context is recognized as an important aspect that influences the drinking behavior of young adults in alcohol research, relatively little work has been conducted in smartphone sensing research on this topic. In this study, we analyze the wee…
While the task of automatically detecting eating events has been examined in prior work using various wearable devices, the use of smartphones as standalone devices to infer eating events remains an open issue. This paper proposes a framework that infers eating vs. non-eating events from passive smartphone sensing and evaluates it on a dataset of 58 college students. First, we show that time of the day and features from modalities such as screen usage, accelerometer, app usage, and location are indicative of eating and non-eating events. Then, we show that eating events can be inferred with an…
The collection of spatiotemporal tourism information is important in smart tourism and user-generated contents are perceived as reliable information. Participatory sensing is a useful method for collecting such data, and the active contribution of users is an important aspect for continuous and efficient data collection. This study has focused on the impact of task allocation interface design and individual personality on data collection efficiency and their contribution in gamified participatory sensing for tourism. We have designed two types of interfaces: a map-based with active selection a…
With the outbreak of the COVID-19 pandemic in 2020, most colleges and universities move to restrict campus activities, reduce indoor gatherings and move instruction online. These changes required that students adapt and alter their daily routines accordingly. To investigate patterns associated with these behavioral changes, we collected smartphone sensing data using the Beiwe platform from two groups of undergraduate students at a major North American university, one from January to March of 2020 (74 participants), the other from May to August (52 participants), to observe the differences in s…
Recognizing surfaces based on their vibration signatures is useful as it can enable tagging of different locations without requiring any additional hardware such as Near Field Communication (NFC) tags. However, previous vibration based surface recognition schemes either use custom hardware for creating and sensing vibration, which makes them difficult to adopt, or use inertial (IMU) sensors in commercial off-the-shelf (COTS) smartphones to sense movements produced due to vibrations, which makes them coarse-grained because of the low sampling rates of IMU sensors. The mainstream COTS smartphone…
We present POD, a smartphone that flies, as a new way to achieve hands-free, eyes-up mobile computing. Unlike existing drone-carried user interfaces, POD features a smartphone-sized display and the computing and sensing power of a modern smartphone. We share our experience in building a prototype of POD, discuss the technical challenges facing it, and describe early results toward addressing them.
Smartphones enable understanding human behavior with activity recognition to support people's daily lives. Prior studies focused on using inertial sensors to detect simple activities (sitting, walking, running, etc.) and were mostly conducted in homogeneous populations within a country. However, people are more sedentary in the post-pandemic world with the prevalence of remote/hybrid work/study settings, making detecting simple activities less meaningful for context-aware applications. Hence, the understanding of (i) how multimodal smartphone sensors and machine learning models could be used t…
There is a research field of human activity recognition that automatically recognizes a user's physical activity through sensing technology incorporated in smartphones and other devices. When sensing daily activity, various measurement conditions, such as device type, possession method, wearing method, and measurement application, are often different depending on the user and the date of the measurement. Models that predict activity from sensor values are often implemented by machine learning and are trained using a large amount of activity-labeled sensor data measured from many users who prov…
Mood inference with mobile sensing data has been studied in ubicomp literature over the last decade. This inference enables context-aware and personalized user experiences in general mobile apps and valuable feedback and interventions in mobile health apps. However, even though model generalization issues have been highlighted in many studies, the focus has always been on improving the accuracies of models using different sensing modalities and machine learning techniques, with datasets collected in homogeneous populations. In contrast, less attention has been given to studying the performance…
