This paper investigates the user experience of visualizations of a machine learning (ML) system that recognizes objects in images. This is important since even good systems can fail in unexpected ways as misclassifications on photo-sharing websites showed. In our study, we exposed users with a background in ML to three visualizations of three systems with different levels of accuracy. In interviews, we explored how the visualization helped users assess the accuracy of systems in use and how the visualization and the accuracy of the system affected trust and reliance. We found that participants do not only focus on accuracy when assessing ML systems. They also take the perceived plausibility and severity of misclassification into account and prefer seeing the probability of predictions. Semantically plausible errors are judged as less severe than errors that are implausible, which means that system accuracy could be communicated through the types of errors.
Interface design can directly influence trustworthiness of a software. Thereby, it affects users' intention to use a tool. Previous research on user trust has not comprehensively addressed user interface design, though. We lack an understanding of what makes interfaces trustworthy (1), as well as actionable measures to improve trustworthiness (2). We contribute to this by addressing both gaps. Based on a systematic literature review, we give a thorough overview over the theory on user trust and provide a taxonomy of factors influencing user interface trustworthiness. Then, we derive concrete m…
The voluntary carbon market is an important building block in the fight against climate change. However, it is not trivial for consumers to verify whether carbon offset projects deliver what they promise. While technical solutions for measuring their impact are emerging, there is a lack of understanding of how to translate this data into interface designs that mediate the establishment of trust. With interaction between users and offset projects mainly happening online, it is critical to meet this design challenge. To this end, we designed and evaluated interfaces with varying trust cues for c…
In this work, a real-time hand gesture recognition system-based human-computer interface (HCI) is presented. The system consists of six stages: (1) hand detection, (2) gesture segmentation, (3) use of five pre-trained convolutional neural network models (CNN) and vision transformer (ViT), (4) building an interactive human-machine interface (HMI), (5) development of a gesture-controlled virtual mouse, (6) use of Kalman filter to estimate the hand position, based on that the smoothness of the motion of pointer is improved. In our work, five pre-trained CNN (VGG16, VGG19, ResNet50, ResNet101, and…
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce costs and interactive friction, supported by the rapid growth in capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a hearing speech 18% WER. Ou…
Emotion recognition technology is crucial in providing a personalized user experience. It is especially important in virtual reality(VR) to assess the user's emotions to enhance their sense of immersion. We propose an emotion recognition interface that incorporates the user's biometric data with machine learning technology for increasing user engagement in VR. Our key technologies include brainwave sensors and eye-tracking cameras embedded in a VR headset, which seamlessly acquire physiological signals, and secondly, an attractiveness recognition algorithm that uses bio-signals to predict the …
Conceptualizing away the sketch processing details in a user interface will enable general users and domain experts to create more complex sketches. There are many domains for which sketch recognition systems are being developed. But they entail image-processing skill if they are to handle the details of each domain, and also they are lengthy to build. The implemented system goal is to enable user interface designers and domain experts who may not have proficiency in sketch recognition to be able to construct these sketch systems. This sketch recognition system takes in rough sketches from use…
Engagement is a key indicator of the quality of learning experience, and one that plays a major role in developing intelligent educational interfaces. Any such interface requires the ability to recognise the level of engagement in order to respond appropriately; however, there is very little existing data to learn from, and new data is expensive and difficult to acquire. This paper presents a deep learning model to improve engagement recognition from images that overcomes the data sparsity challenge by pre-training on readily available basic facial expression data, before training on specialis…
Body actions and head gestures are natural interfaces for interaction in virtual environments. Existing methods for in-place body action recognition often require hardware more than a head-mounted display (HMD), making body action interfaces difficult to be introduced to ordinary virtual reality (VR) users as they usually only possess an HMD. In addition, there lacks a unified solution to recognize in-place body actions and head gestures. This potentially hinders the exploration of the use of in-place body actions and head gestures for novel interaction experiences in virtual environments. We …
External Human-Machine Interface based on Intent Recognition: Framework Design and Experimental Validation
College of Transportation and the Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University（同济大学交通学院和道路与交通工程教育部重点实验室）；Department of Civil and Environmental Engineering, National University of Singapore（新加坡国立大学土木与环境工程系）
User interfaces provide an interactive window between physical and virtual environments. A new concept in the field of human-computer interaction is a soft user interface; a compliant surface that facilitates touch interaction through deformation. Despite the potential of these interfaces, they currently lack a signal processing framework that can efficiently extract information from their deformation. Here we present OrbTouch, a device that uses statistical learning algorithms, based on convolutional neural networks, to map deformations from human touch to categorical labels (i.e., gestures) …
Capacitive sensing is a prominent technology that is cost-effective and low power consuming with fast recognition speed compared to existing sensing systems. On account of these advantages, Capacitive sensing has been widely studied and commercialized in the domains of touch sensing, localization, existence detection, and contact sensing interface application such as human-computer interaction. However, as a non-contact proximity sensing scheme is easily affected by the disturbance of peripheral objects or surroundings, it requires considerable sensitive data processing than contact sensing, l…
In the past decade, the financial industry has experienced a technology revolution. While we witness a rapid introduction of conversational bots for financial services, there is a lack of understanding of conversational user interfaces (CUI) features in this domain. The finance industry also deals with highly sensitive information and monetary transactions, presenting a challenge for developers and financial providers. Through a study on how to design text-based conversational financial interfaces with N=410 participants, we outline user requirements of trustworthy CUI design for financial bot…
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce device costs and interactive friction. This transition has been hastened by the increasing capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a …
Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents
University of Notre Dame（圣约翰大学）；Northeastern University（东北大学）；Virginia Tech（弗吉尼亚理工大学）；University of Washington（华盛顿大学）；Johns Hopkins University（约翰霍普金斯大学）
Fraunhofer IAIS（弗劳恩霍夫智能分析与信息系统研究所）；University of Bonn（波恩大学）
Applications which use human speech as an input require a speech interface with high recognition accuracy. The words or phrases in the recognised text are annotated with a machine-understandable meaning and linked to knowledge graphs for further processing by the target application. These semantic annotations of recognised words can be represented as a subject-predicate-object triples which collectively form a graph often referred to as a knowledge graph. This type of knowledge representation facilitates to use speech interfaces with any spoken input application, since the information is repre…
Nowadays, autonomous mobile robots support people in many areas where human presence either redundant or too dangerous. They have successfully proven themselves in expeditions, gas industry, mines, warehouses, etc. However, even legged robots may stuck in rough terrain conditions requiring human cognitive abilities to navigate the system. While gamepads and keyboards are convenient for wheeled robot control, the quadruped robot in 3D space can move along all linear coordinates and Euler angles, requiring at least 12 buttons for independent control of their DoF. Therefore, more convenient inter…
Complex robotic tasks require human collaboration to benefit from their high dexterity. Frequent human-robot interaction is mentally demanding and time-consuming. Intuitive and easy-to-use robot control interfaces reduce the negative influence on workers, especially inexperienced users. In this paper, we present CobotTouch, a novel intuitive robot control interface with fingertip haptic feedback. The proposed interface consists of a projected Graphical User Interface on the robotic arm to control the position of the robot end-effector based on gesture recognition, and a wearable haptic interfa…
George Mason University（乔治梅森大学）；Duke University（杜克大学）
Past studies have illustrated the prevalence of UI dark patterns, or user interfaces that can lead end-users toward (unknowingly) taking actions that they may not have intended. Such deceptive UI designs can result in adverse effects on end users, such as oversharing personal information or financial loss. While significant research progress has been made toward the development of dark pattern taxonomies, developers and users currently lack guidance to help recognize, avoid, and navigate these often subtle design motifs. However, automated recognition of dark patterns is a challenging task, as…
Brain-Computer Interface (BCI) is a system empowering humans to communicate with or control the outside world with exclusively brain intentions. Electroencephalography (EEG) based BCIs are promising solutions due to their convenient and portable instruments. Motor imagery EEG (MI-EEG) is a kind of most widely focused EEG signals, which reveals a subjects movement intentions without actual actions. Despite the extensive research of MI-EEG in recent years, it is still challenging to interpret EEG signals effectively due to the massive noises in EEG signals (e.g., low signal noise ratio and incom…
Touch,face,voice recognition and movement sensors all are part of an emerging field of computing often called natural user interface, or NUI. Interacting with technology in these humanistic ways is no longer limited to high tech secret agents. Gesture Touch, face, voice recognition and movement sensors all are part of an emerging field of computing often called natural user interface, or NUI. Interacting with technology in these humanistic ways is no longer limited to high tech secret agents. Gesture recognition is the process by which gestures formed by a user are made known to the system. In…
Speech recognition is a prominent technology, which helps us to develop a Natural language interface through speech for the Human-Robot Interaction (HRI). It allows the computer to take the spoken instructions, interpret it, and generate text from it. In this paper, we propose a phoneme based speech recognition system to control industrial robots. Speech recognition has become one of the popular interfaces when it comes to reducing robot operator's efforts to control and command the robot. This paper intends to investigate the potential of Linear Predictive coding technique to develop a stable…
Co-Authoring the Self: A Human-AI Interface for Interest Reflection in Recommenders
Grouplens Research, University of Minnesota（Grouplens研究院、明尼苏达大学）；Department of Computer Science and Engineering, University of Minnesota（计算机科学与工程系、明尼苏达大学）
City, University of London（伦敦城市大学）；Fujitsu Laboratories of Europe（富士通欧洲实验室）；Intesa Sanpaolo S.p.A.（意大利联合圣保罗银行）
With Artificial intelligence (AI) to aid or automate decision-making advancing rapidly, a particular concern is its fairness. In order to create reliable, safe and trustworthy systems through human-centred artificial intelligence (HCAI) design, recent efforts have produced user interfaces (UIs) for AI experts to investigate the fairness of AI models. In this work, we provide a design space exploration that supports not only data scientists but also domain experts to investigate AI fairness. Using loan applications as an example, we held a series of workshops with loan officers and data scienti…
Nowadays, speech is becoming a more common, if not standard, interface to technology. This can be seen in the trend of technology changes over the years. Increasingly, voice is used to control programs, appliances and personal devices within homes, cars, workplaces, and public spaces through smartphones and home assistant devices using Amazon's Alexa, Google's Assistant and Apple's Siri, and other proliferating technologies. However, most speech interfaces are not accessible for Deaf and Hard-of-Hearing (DHH) people. In this paper, performances of current Automatic Speech Recognition (ASR) wit…
Laval University（拉瓦尔大学）；Université du Québec à Montréal（魁北克大学蒙特利尔分校）
This paper presents a control interface to translate the residual body motions of individuals living with severe disabilities, into control commands for body-machine interaction. A custom, wireless, wearable multi-sensor network is used to collect motion data from multiple points on the body in real-time. The solution proposed successfully leverage electromyography gesture recognition techniques for the recognition of inertial measurement units-based commands (IMU), without the need for cumbersome and noisy surface electrodes. Motion pattern recognition is performed using a computationally ine…
Emotion-Aware Interaction Design in Intelligent User Interface Using Multi-Modal Deep Learning
Carnegie Mellon University（卡内基梅隆大学）；University of Maryland（马里兰大学）；School of Visual Arts（视觉艺术学院）；New York University（纽约大学）；University of Southern California（南加州大学）
FMCW radar could detect object's range, speed and Angleof-Arrival, advantages are robust to bad weather, good range resolution, and good speed resolution. In this paper, we consider the FMCW radar as a novel interacting interface on laptop. We merge sequences of object's range, speed, azimuth information into single input, then feed to a convolution neural network to learn spatial and temporal patterns. Our model achieved 96% accuracy on test set and real-time test.
A Brain-Computer Interface Augmented Reality Framework with Auto-Adaptive SSVEP Recognition
Missouri University of Science and Technology（密苏里科技大学）；Ain Shams University（艾因夏姆斯大学）；The American University in Cairo（开罗美国大学）；German University in Cairo（开罗德国大学）
MIT（麻省理工学院）；Harvard Graduate School of Design（哈佛大学设计研究生院）；Harvard University（哈佛大学）
Vast improvements in natural language understanding and speech recognition have paved the way for conversational interaction with computers. While conversational agents have often been used for short goal-oriented dialog, we know little about agents for developing computer programs. To explore the utility of natural language for programming, we conducted a study ($n$=45) comparing different input methods to a conversational programming system we developed. Participants completed novice and advanced tasks using voice-based, text-based, and voice-or-text-based systems. We found that users apprec…
