Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning
Australian National University（澳大利亚国立大学）；Monash University（莫纳什大学）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织数据61部）；University of New South Wales（新南威尔士大学）；Shanghai Jiao Tong University（上海交通大学）；Fujian University of Technology（福建理工大学）
According to the World Health Organization(WHO), it is estimated that approximately 1.3 billion people live with some forms of vision impairment globally, of whom 36 million are blind. Due to their disability, engaging these minority into the society is a challenging problem. The recent rise of smart mobile phones provides a new solution by enabling blind users' convenient access to the information and service for understanding the world. Users with vision impairment can adopt the screen reader embedded in the mobile operating systems to read the content of each screen within the app, and use gestures to interact with the phone. However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app. Unfortunately, more than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. Most of these issues are caused by developers' lack of awareness and knowledge in considering the minority. And even if developers want to add the labels to UI components, they may not come up with concise and clear description as most of them are of no visual issues. To overcome these challenges, we develop a deep-learning based model, called LabelDroid, to automatically predict the labels of image-based buttons by learning from large-scale commercial apps in Google Play. The experimental results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers.
Smartphones are a crucial aspect of routine life in the modern world, and viewing information graphics such as charts becomes common practice for many unassuming tasks. However, for the vision impaired, accessing graphical material presents many difficulties. Android smartphones usually come preinstalled with Google Talkback as a default screen-reader, which attempts to cater for the visually impaired by providing supplementary auditory information when interfacing with supported applications. Still, the crux of this situation is that screen-readers rely on developers correctly implementing th…
Mobile apps are indispensable for people's daily life. Complementing with automated GUI testing, manual testing is the last line of defence for app quality. However, the repeated actions and easily missing of functionalities make manual testing time-consuming and inefficient. Inspired by the game candy crush with flashy candies as hint moves for players, we propose an approach named NaviDroid for navigating testers via highlighted next operations for more effective and efficient testing. Within NaviDroid, we construct an enriched state transition graph with the triggering actions as the edges …
Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps
The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）；The Hong Kong University of Science and Technology（香港科学大学）
Less Cybersickness, Please: Demystifying and Detecting Stereoscopic Visual Inconsistencies in Virtual Reality Apps
The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；Research Institute of Trustworthy Autonomous Systems（可信自主系统研究院）；Southern University of Science and Technology（南方科技大学）
Mobile devices, specifically, smartphones proved easy and quick access to data visualisations throughout various tracking apps. Mobile health (mHealth) apps have given non-expert users access to data visualisation to track their activities and health-related issues such as heart tracking and medication. However, no work is done on user experience or perception of data visualisations in mHealth apps. App reviews offer an indirect anchor for researchers to examine how non-expert users perceive and interact with data visualisations and identify the key challenges and recommendations. This paper i…
The paper discusses the experience of producing and distributing an iPhone app for promotion of the Maltese Cultural Heritage on behalf of the Malta Tourism Authority. Thanks to its position at the heart of the Mediterranean Sea, Malta has been a crossroads of civilisations whose traces are still visible today, leaving a particularly rich and varied cultural heritage, from megalithic temples to baroque palaces and Caravaggio masterpieces. Conveying all these different aspects within a single application, using textual, visual, and audio means, has raised many different issues about the plannin…
Augmented reality (AR) is an emerging technology in mobile app design during recent years. However, usability challenges in these apps are prominent. There are currently no established guidelines for designing and evaluating interactions in AR as there are in traditional user interfaces. In this work, we aimed to examine the usability of current mobile AR applications and interpreting classic usability heuristics in the context of mobile AR. Particularly, we focused on AR home design apps because of their popularity and ability to incorporate important mobile AR interaction schemas. Our findin…
Supporting voice commands in applications presents significant benefits to users. However, adding such support to existing GUI-based web apps is effort-consuming with a high learning barrier, as shown in our formative study, due to the lack of unified support for creating multimodal interfaces. We present Geno---a developer tool for adding the voice input modality to existing web apps without requiring significant NLP expertise. Geno provides a high-level workflow for developers to specify functionalities to be supported by voice (intents), create language models for detecting intents and the …
The industry for children's apps is thriving at the cost of children's privacy: these apps routinely disclose children's data to multiple data trackers and ad networks. As children spend increasing time online, such exposure accumulates to long-term privacy risks. In this paper, we used a mixed-methods approach to investigate why this is happening and how developers might change their practices. We base our analysis against 5 leading data protection frameworks that set out requirements and recommendations for data collection in children's apps. To understand developers' perspectives and constr…
Personal data visualisations are becoming a critical contributor toward the successful adoption of mobile health (m-health) apps. Thus, understanding user needs and challenges when using mobile personal data visualisation is essential to ensuring the adoption of these apps. This paper presents the results of a user survey to understand users' demographics, tasks, needs, and challenges of using mobile personal data visualisations. We had 56 complete responses. The survey's key findings are: 1) 51\% of the users use multiple health tracking apps to achieve their goals/needs; 2) bar charts and pi…
How Should We Support Designing Privacy-Friendly Apps for Children? Using a Research through Design Process to Understand Developers' Needs and Challenges
Due to the frequent encountering of web URLs in various application scenarios (e.g., chatting and email reading), many mobile apps build their in-app browsing interfaces (IABIs) to provide a seamless user experience. Although this achieves user-friendliness by avoiding the constant switching between the subject app and the system built-in browser apps, we find that IABIs, if not well designed or customized, could result in usability security risks. In this paper, we conduct the first empirical study on the usability (in)security of in-app browsing interfaces in both Android and iOS apps. Speci…
Obesity is known to lower the quality of life substantially. It is often associated with increased chances of non-communicable diseases such as diabetes, cardiovascular problems, various cancers, etc. Evidence suggests that diet-related mobile applications play a vital role in assisting individuals in making healthier choices and keeping track of food intake. However, due to an abundance of similar applications, it becomes pertinent to evaluate each of them in terms of functionality, usability, and possible design issues to truly determine state-of-the-art solutions for the future. Since these…
Mobile mental health applications are seen as a promising way to fulfill the growing need for mental health care. Although there are more than ten thousand mental health apps available on app marketplaces, such as Google Play and Apple App Store, many of them are not evidence-based, or have been minimally evaluated or regulated. The real-life experience and concerns of the app users are largely unknown. To address this knowledge gap, we analyzed 2159 user reviews from 117 Android apps and 2764 user reviews from 76 iOS apps. Our findings include the critiques around inconsistent moderation stan…
We present Project IRL (In Real Life), a suite of five mobile apps we created to explore novel ways of supporting in-person social interactions with augmented reality. In recent years, the tone of public discourse surrounding digital technology has become increasingly critical, and technology's influence on the way people relate to each other has been blamed for making people feel "alone together," diverting their attention from truly engaging with one another when they interact in person. Motivated by this challenge, we focus on an under-explored design space: playful co-located interactions.…
Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multi-modal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to…
