In this paper, we investigate common pitfalls affecting the evaluation of authentication systems based on touch dynamics. We consider different factors that lead to misrepresented performance, are incompatible with stated system and threat models or impede reproducibility and comparability with previous work. Specifically, we investigate the effects of (i) small sample sizes (both number of users and recording sessions), (ii) using different phone models in training data, (iii) selecting non-contiguous training data, (iv) inserting attacker samples in training data and (v) swipe aggregation. We perform a systematic review of 30 touch dynamics papers showing that all of them overlook at least one of these pitfalls. To quantify each pitfall's effect, we design a set of experiments and collect a new longitudinal dataset of touch interactions from 515 users over 31 days comprised of 1,194,451 unique strokes. Part of this data is collected in-lab with Android devices and the rest remotely with iOS devices, allowing us to make in-depth comparisons. We make this dataset and our code available online. Our results show significant percentage-point changes in reported mean EER for several pitfalls: including attacker data (2.55%), non-contiguous training data (3.8%) and phone model mixing (3.2%-5.8%). We show that, in a common evaluation setting, the cumulative effects of these evaluation choices result in a combined difference of 8.9% EER. We also largely observe these effects across the entire ROC curve. The pitfalls are evaluated on four distinct classifiers - SVM, Random Forest, Neural Network, and kNN. Furthermore, we explore additional considerations for fair evaluation when building touch-based authentication systems and quantify their impacts. Based on these insights, we propose a set of best practices that, will lead to more realistic and comparable reporting of results in the field.
