arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.00201cs.CVcs.CL
收藏

DreamStruct:通过合成数据生成理解幻灯片与用户界面

DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

Carnegie Mellon University(卡内基梅隆大学);University of Texas Austin(德克萨斯大学奥斯汀分校);Aalto University(阿尔托大学)

Yi-Hao Peng, Faria Huq, Yue Jiang, Jason Wu, Amanda Xin Yue Li, Jeffrey Bigham, Amy Pavel

AI总结:

DreamStruct 提出利用代码生成合成带标签的结构化视觉数据,以少量人工标注即可训练模型,并在幻灯片与UI的视觉元素识别、内容描述和类型分类三项任务上取得性能提升。

摘要:

使机器能够理解幻灯片和用户界面等结构化视觉内容,对于让残障人士能够访问这些内容至关重要。然而,在计算上实现这种理解此前需要人工收集和标注数据,既耗时又费力。为克服这一挑战,我们提出了一种利用代码生成来生成带有目标标签的合成结构化视觉内容的方法。我们的方法允许人们创建具有内置标签的数据集,并仅使用少量人工标注的示例来训练模型。我们在理解幻灯片和用户界面的三项任务中展示了性能提升:识别视觉元素、描述视觉内容以及对视觉内容类型进行分类。

英文摘要:

Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑