猜猜我的屏幕上有什么？基于主动学习的智能手机屏幕截图聚类
Guess What's on my Screen? Clustering Smartphone Screenshots with Active Learning
摘要：
个人日常活动的很大一部分是通过数字设备体验的。特别是智能手机已经成为内容消费和社交互动的首选界面之一。因此，识别嵌入在频繁捕获的智能手机屏幕截图中的内容是研究媒体行为和健康干预计划的关键先决条件，这些计划分析了活动的相互作用和随着时间的推移的内容切换。屏幕截图图像可以描述异构的内容和应用程序，使得适当分类的先验定义成为一项繁琐的任务，即使对于人类来说也是如此。对屏幕上捕获的敏感数据的隐私保护意味着与手动注释相关的成本很大，因为这项工作无法众包。因此，有必要研究数字屏幕截图分类的无监督和半监督方法的实用性。这项工作介绍了在只有有限数量的标签可用时，在大型屏幕截图集上应用聚类的含义。在本文中，我们开发了一个框架，用于将K-Means聚类与主动学习相结合，以有效利用标记和未标记的样本，其目标是发现潜在类并描述大量的屏幕截图数据。我们测试了类概率传播的SVM嵌入式或XGBoost嵌入式解决方案是否提供了更良好的集群配置。视觉和文本矢量表示的屏幕截图图像的推导和组合，以评估多模态功能的整体性能的相对贡献。
英文摘要：
A significant proportion of individuals' daily activities is experienced through digital devices. Smartphones in particular have become one of the preferred interfaces for content consumption and social interaction. Identifying the content embedded in frequently-captured smartphone screenshots is thus a crucial prerequisite to studies of media behavior and health intervention planning that analyze activity interplay and content switching over time. Screenshot images can depict heterogeneous contents and applications, making the a priori definition of adequate taxonomies a cumbersome task, even for humans. Privacy protection of the sensitive data captured on screens means the costs associated with manual annotation are large, as the effort cannot be crowd-sourced. Thus, there is need to examine utility of unsupervised and semi-supervised methods for digital screenshot classification. This work introduces the implications of applying clustering on large screenshot sets when only a limited amount of labels is available. In this paper we develop a framework for combining K-Means clustering with Active Learning for efficient leveraging of labeled and unlabeled samples, with the goal of discovering latent classes and describing a large collection of screenshot data. We tested whether SVM-embedded or XGBoost-embedded solutions for class probability propagation provide for more well-formed cluster configurations. Visual and textual vector representations of the screenshot images are derived and combined to assess the relative contribution of multi-modal features to the overall performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
