On-device machine learning (ML) is quickly gaining popularity among mobile apps. It allows offline model inference while preserving user privacy. However, ML models, considered as core intellectual properties of model owners, are now stored on billions of untrusted devices and subject to potential thefts. Leaked models can cause both severe financial loss and security consequences. This paper presents the first empirical study of ML model protection on mobile devices. Our study aims to answer three open questions with quantitative evidence: How widely is model protection used in apps? How robust are existing model protection techniques? What impacts can (stolen) models incur? To that end, we built a simple app analysis pipeline and analyzed 46,753 popular apps collected from the US and Chinese app markets. We identified 1,468 ML apps spanning all popular app categories. We found that, alarmingly, 41% of ML apps do not protect their models at all, which can be trivially stolen from app packages. Even for those apps that use model protection or encryption, we were able to extract the models from 66% of them via unsophisticated dynamic analysis techniques. The extracted models are mostly commercial products and used for face recognition, liveness detection, ID/bank card recognition, and malware detection. We quantitatively estimated the potential financial and security impact of a leaked model, which can amount to millions of dollars for different stakeholders. Our study reveals that on-device models are currently at high risk of being leaked; attackers are highly motivated to steal such models. Drawn from our large-scale study, we report our insights into this emerging security problem and discuss the technical challenges, hoping to inspire future research on robust and practical model protection for mobile devices.
Google and Apple have jointly provided an API for exposure notification in order to implement decentralized contract tracing apps using Bluetooth Low Energy, the so-called "Google/Apple Proposal", which we abbreviate by "GAP". We demonstrate that in real-world scenarios the current GAP design is vulnerable to (i) profiling and possibly de-anonymizing infected persons, and (ii) relay-based wormhole attacks that basically can generate fake contacts with the potential of affecting the accuracy of an app-based contact tracing system. For both types of attack, we have built tools that can easily be…
With smartphone technologies enhanced way of interacting with the world around us, it has also been paving the way for easier access to our private and personal information. This has been amplified by the existence of numerous embedded sensors utilized by millions of apps to users. While mobile apps have positively transformed many aspects of our lives with new functionalities, many of these applications are taking advantage of vast amounts of data, privacy apps, a form of Privacy Enhancing Technology can be an effective privacy management tool for smartphones. To protect against vulnerabiliti…
Privacy in apps is a topic of widespread interest because many apps collect and share large amounts of highly sensitive information. In response, China introduced a range of new data protection laws over recent years, notably the Personal Information Protection Law (PIPL) in 2021. So far, there exists limited research on the impacts of these new laws on apps' privacy practices. To address this gap, this paper analyses data collection in pairs of 634 Chinese iOS apps, one version from early 2020 and one from late 2021. Our work finds that many more apps now implement consent. Yet, those end-use…
Data protection law, including the General Data Protection Regulation (GDPR), usually requires a privacy policy before data can be collected from individuals. We analysed 15,145 privacy policies from 26,910 mobile apps in May 2019 (about one year after the GDPR came into force), finding that only opening the policy webpages shares data with third-parties for 48.5% of policies, potentially violating the GDPR. We compare this data sharing across countries, payment models (free, in-app-purchases, paid) and platforms (Google Play Store, Apple App Store). We further contacted 52 developers of apps,…
The digital age has brought a world of opportunity to children. Connectivity can be a game-changer for some of the world's most marginalized children. However, while legislatures around the world have enacted regulations to protect children's online privacy, and app stores have instituted various protections, privacy in mobile apps remains a growing concern for parents and wider society. In this paper, we explore the potential privacy issues and threats that exist in these apps. We investigate 20,195 mobile apps from the Google Play store that are designed particularly for children (Family app…
The purpose of the General Data Protection Regulation (GDPR) is to provide improved privacy protection. If an app controls personal data from users, it needs to be compliant with GDPR. However, GDPR lists general rules rather than exact step-by-step guidelines about how to develop an app that fulfills the requirements. Therefore, there may exist GDPR compliance violations in existing apps, which would pose severe privacy threats to app users. In this paper, we take mobile health applications (mHealth apps) as a peephole to examine the status quo of GDPR compliance in Android apps. We first pro…
Nowadays many people store photos in smartphones. Many of the photos contain sensitive, private information, such as a photocopy of driver's license and credit card. An arising privacy concern is with the unauthorized accesses to such private photos by installed apps. Coarse-grained access control systems such as the Android permission system offer all-or-nothing access to photos stored on smartphones, and users are unaware of the exact behavior of installed apps. Our analysis finds that 82% of the top 200 free apps in a popular Android app store have complete access to stored photos and netwo…
The recent SARS-CoV-2 pandemic gave rise to management approaches using mobile apps for contact tracing. The corresponding apps track individuals and their interactions, to facilitate alerting users of potential infections well before they become infectious themselves. Naive implementation obviously jeopardizes the privacy of health conditions, location, activities, and social interaction of its users. A number of protocol designs for colocation tracking have already been developed, most of which claim to function in a privacy preserving manner. However, despite claims such as "GDPR compliance…
Third-party security apps are an integral part of the Android app ecosystem. Many users install them as an extra layer of protection for their devices. There are hundreds of such security apps, both free and paid in Google Play Store and some of them are downloaded millions of times. By installing security apps, the smartphone users place a significant amount of trust towards the security companies who developed these apps, because a fully functional mobile security app requires access to many smartphone resources such as the storage, text messages and email, browser history, and information a…
Contact tracing is an essential tool for public health officials and local communities to fight the spread of novel diseases, such as for the COVID-19 pandemic. The Singaporean government just released a mobile phone app, TraceTogether, that is designed to assist health officials in tracking down exposures after an infected individual is identified. However, there are important privacy implications of the existence of such tracking apps. Here, we analyze some of those implications and discuss ways of ameliorating the privacy concerns without decreasing usefulness to public health. We hope in w…
Contemporary mobile applications (apps) are designed to track, use, and share users' data, often without their consent, which results in potential privacy and transparency issues. To investigate whether mobile apps have always been (non-)transparent regarding how they collect information about users, we perform a longitudinal analysis of the historical versions of 268 Android apps. These apps comprise 5,240 app releases or versions between 2008 and 2016. We detect inconsistencies between apps' behaviors and the stated use of data collection in privacy policies to reveal compliance issues. We u…
Mobile databases are the statutory backbones of many applications on smartphones, and they store a lot of sensitive information. However, vulnerabilities in the operating system or the app logic can lead to sensitive data leakage by giving the adversaries unauthorized access to the app's database. In this paper, we study such vulnerabilities to define a threat model, and we propose an OS-version independent protection mechanism that app developers can utilize to detect such attacks. To do so, we model the user behavior with the database query workload created by the original apps. Here, we mod…
The General Data Protection Regulation (GDPR) aims to ensure that all personal data processing activities are fair and transparent for the European Union (EU) citizens, regardless of whether these are carried out within the EU or anywhere else. To this end, it sets strict requirements to transfer personal data outside the EU. However, checking these requirements is a daunting task for supervisory authorities, particularly in the mobile app domain due to the huge number of apps available and their dynamic nature. In this paper, we propose a fully automated method to assess the compliance of mob…
Ample research has demonstrated that compliance with data protection principles remains limited on the web and mobile. For example, almost none of the apps on the Google Play Store fulfil the minimum requirements regarding consent under EU and UK law, while most of them share tracking data with companies like Google/Alphabet and Facebook/Meta and would likely need to seek consent from their users. Indeed, recent privacy efforts and enforcement by Apple have had - in some regards - a more pronounced effect on apps' data practices than the EU's ambitious General Data Protection Regulation (GDPR)…
THEMIS: Towards Practical Intellectual Property Protection for Post-Deployment On-Device Deep Learning Models
The University of Melbourne（墨尔本大学）；The University of Western Australia（西澳大学）；City University of Hong Kong（香港城市大学）；Technical University of Munich（慕尼黑工业大学）
Voice-controlled smart speaker devices have gained a foothold in many modern households. Their prevalence combined with their intrusion into core private spheres of life has motivated research on security and privacy intrusions, especially those performed by third-party applications used on such devices. In this work, we take a closer look at such third-party applications from a less pessimistic angle: we consider their potential to provide personalized and secure capabilities and investigate measures to authenticate users (``PIN'', ``Voice authentication'', ``Notification'', and presence of `…
SAP Security Research, SAP Global Security（SAP全球安全事业部SAP安全研究院）；Hasso Plattner Institute, University of Potsdam（波茨坦大学哈索·普拉特纳研究所）；SAP Portfolio Strategy & Technology Adoption, SAP SE（SAP SE SAP产品组合战略与技术采纳部）；Logistic & IT Services, City of Antibes Juan-les-Pins（昂蒂布-朱安雷宾市物流与信息技术服务部）
Current developments in Enterprise Systems observe a paradigm shift, moving the needle from the backend to the edge sectors of those; by distributing data, decentralizing applications and integrating novel components seamlessly to the central systems. Distributively deployed AI capabilities will thrust this transition. Several non-functional requirements arise along with these developments, security being at the center of the discussions. Bearing those requirements in mind, hereby we propose an approach to holistically protect distributed Deep Neural Network (DNN) based/enhanced software asset…
Recent studies observe that app foreground is the most striking component that influences the access control decisions in mobile platform, as users tend to deny permission requests lacking visible evidence. However, none of the existing permission models provides a systematic approach that can automatically answer the question: Is the resource access indicated by app foreground? In this work, we present the design, implementation, and evaluation of COSMOS, a context-aware mediation system that bridges the semantic gap between foreground interaction and background access, in order to protect sy…
Cross-app collaboration via inter-component communication is a fundamental mechanism on Android. Although it brings the benefits such as functionality reuse and data sharing, a threat called component hijacking is also introduced. By hijacking a vulnerable component in victim apps, an attack app can escalate its privilege for originally prohibited operations. Many prior studies have been performed to understand and mitigate this issue, but component hijacking remains a serious open problem in the Android ecosystem due to no effective defense deployed in the wild. In this paper, we present our …
Many Android applications collect data from users. When they do, they must protect this collected data according to the current legal frameworks. Such data protection has become even more important since the European Union rolled out the General Data Protection Regulation (GDPR). App developers have limited tool support to reason about data protection throughout their app development process. Although many Android applications state a privacy policy, privacy policy compliance checks are currently manual, expensive, and prone to error. One of the major challenges in privacy audits is the signif…
Data protection regulations generally afford individuals certain rights over their personal data, including the rights to access, rectify, and delete the data held on them. Exercising such rights naturally requires those with data management obligations (service providers) to be able to match an individual with their data. However, many mobile apps collect personal data, without requiring user registration or collecting details of a user's identity (email address, names, phone number, and so forth). As a result, a user's ability to exercise their rights will be hindered without means for an in…
Android allows apps to communicate with its system services via system service helpers so that these apps can use various functions provided by the system services. Meanwhile, the system services rely on their service helpers to enforce security checks for protection. Unfortunately, the security checks in the service helpers may be bypassed via directly exploiting the non-SDK (hidden) APIs, degrading the stability and posing severe security threats such as privilege escalation, automatic function execution without users' interactions, crashes, and DoS attacks. Google has proposed various appro…
On modern operating systems, applications under the same user are separated from each other, for the purpose of protecting them against malware and compromised programs. Given the complexity of today's OSes, less clear is whether such isolation is effective against different kind of cross-app resource access attacks (called XARA in our research). To better understand the problem, on the less-studied Apple platforms, we conducted a systematic security analysis on MAC OS~X and iOS. Our research leads to the discovery of a series of high-impact security weaknesses, which enable a sandboxed malici…
A new development of smart-home systems is to use mobile apps to control IoT devices across a Home Area Network (HAN). Those systems tend to rely on the Wi-Fi router to authenticate other devices; as verified in our study, IoT vendors tend to trust all devices connected to the HAN. This treatment exposes them to the attack from malicious apps, particularly those running on authorized phones, which the router does not have information to control, as confirmed in our measurement study. Mitigating this threat cannot solely rely on IoT manufacturers, which may need to change the hardware on the de…
Since SARS-CoV-2 started spreading in Europe in early 2020, there has been a strong call for technical solutions to combat or contain the pandemic, with contact tracing apps at the heart of the debates. The EU's General Daten Protection Regulation (GDPR) requires controllers to carry out a data protection impact assessment (DPIA) where their data processing is likely to result in a high risk to the rights and freedoms (Art. 35 GDPR). A DPIA is a structured risk analysis that identifies and evaluates possible consequences of data processing relevant to fundamental rights and describes the measu…
In recent years the PC has been replaced by mobile devices for many security sensitive operations, both from a privacy and a financial standpoint. While security mechanisms are deployed at various levels, these are frequently put under strain by previously unseen malware. An additional protection layer capable of novelty detection is therefore needed. In this work we propose SpotCheck, an anomaly detector intended to run on Android devices. It samples app executions and submits suspicious apps to more thorough processing by malware sandboxes. We compare Kernel Principal Component Analysis (KPC…
The Processing goes far beyond "the app" -- Privacy issues of decentralized Digital Contact Tracing using the example of the German Corona-Warn-App (CWA)
Home security systems have become increasingly popular since they provide an additional layer of protection and peace of mind. These systems typically include battery-powered motion sensors, contact sensors, and smart locks. Z-Wave is a very popular wireless communication technology for these low-power systems. In this paper, we demonstrate two new attacks targeting Z-Wave devices. First, we show how an attacker can remotely attack Z-Wave security devices to increase their power consumption by three orders of magnitude, reducing their battery life from a few years to just a few hours. Second, …
In this study, we propose a novel SeqMF model to solve the problem of predicting the next app launch during mobile device usage. Although this problem can be represented as a classical collaborative filtering problem, it requires proper modification since the data are sequential, the user feedback is distributed among devices and the transmission of users' data to aggregate common patterns must be protected against leakage. According to such requirements, we modify the structure of the classical matrix factorization model and update the training procedure to sequential learning. Since the data…
Currently, Android malware detection is mostly performed on server side against the increasing number of malware. Powerful computing resource provides more exhaustive protection for app markets than maintaining detection by a single user. However, apart from the applications provided by the official market, apps from unofficial markets and third-party resources are always causing serious security threats to end-users. Meanwhile, it is a time-consuming task if the app is downloaded first and then uploaded to the server side for detection, because the network transmission has a lot of overhead. …
Previous work has investigated the particularities of security practices within specific user communities defined based on country of origin, age, prior tech abuse, and economic status. Their results highlight that current security solutions that adopt a one-size-fits-all-users approach ignore the differences and needs of particular user communities. However, those works focus on a single community or cluster users into hard-to-interpret sub-populations. In this work, we perform a large-scale quantitative analysis of the risk of encountering malware and other potentially unwanted applications …
