学习从图像推断用户界面属性
Learning to Infer User Interface Attributes from Images
摘要:
我们探索了一个新的学习领域,以推断用户界面属性,帮助开发人员自动化的过程中,用户界面的实现。具体地说,给定一个由设计师创建的输入图像,我们学习推断它的实现,当渲染时,在视觉上看起来与输入图像相同。为了实现这一点,我们采用黑盒渲染引擎和它支持的一组属性(例如,颜色、边界半径、阴影或文本属性),使用它来生成合适的合成训练数据集,然后训练专门的神经模型来预测每个属性值。为了提高像素级精度,我们还使用模仿学习来训练神经策略,该神经策略通过学习计算原始图像和渲染图像在其属性空间中的相似性来细化预测属性值,而不是基于像素值的差异。我们实例化了我们的方法来推断Android Button属性值,并在由真实世界的Google Play Store应用程序组成的数据集上实现了92.5%的准确率。
英文摘要:
We explore a new domain of learning to infer user interface attributes that helps developers automate the process of user interface implementation. Concretely, given an input image created by a designer, we learn to infer its implementation which when rendered, looks visually the same as the input image. To achieve this, we take a black box rendering engine and a set of attributes it supports (e.g., colors, border radius, shadow or text properties), use it to generate a suitable synthetic training dataset, and then train specialized neural models to predict each of the attribute values. To improve pixel-level accuracy, we additionally use imitation learning to train a neural policy that refines the predicted attribute values by learning to compute the similarity of the original and rendered images in their attribute space, rather than based on the difference of pixel values. We instantiate our approach to the task of inferring Android Button attribute values and achieve 92.5% accuracy on a dataset consisting of real-world Google Play Store applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。