ScreenQA：面向移动应用截屏的大规模问答对数据集
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots
AI总结：
ScreenQA 在 RICO 数据集上构建了 86k 个移动截屏问答对，用于基准测试屏幕阅读理解能力，并通过零样本、微调和迁移学习实验验证了其在移动及 Web 应用中的有效性。
摘要：
我们提出了 ScreenQA，这是一个新颖的基准数据集，旨在通过问答推进屏幕内容理解。现有的屏幕数据集要么侧重于低层级的结构和组件理解，要么侧重于更高层级的复合任务，例如面向自主智能体的导航和任务完成。ScreenQA 试图弥合这一差距。通过在 RICO 数据集上标注 86k 个问答对，我们旨在对屏幕阅读理解能力进行基准测试，从而为基于视觉的截屏自动化奠定基础。我们的标注涵盖完整答案、简短答案短语以及带边界框的相应 UI 内容，支持四个子任务以应对各种应用场景。我们在零样本、微调和迁移学习设置下，使用开放权重模型和专有模型评估了该数据集的有效性。我们进一步展示了向 Web 应用的正向迁移，突显了其在移动应用之外的潜力。
英文摘要：
We introduce ScreenQA, a novel benchmarking dataset designed to advance screen content understanding through question answering. The existing screen datasets are focused either on low-level structural and component understanding, or on a much higher-level composite task such as navigation and task completion for autonomous agents. ScreenQA attempts to bridge this gap. By annotating 86k question-answer pairs over the RICO dataset, we aim to benchmark the screen reading comprehension capacity, thereby laying the foundation for vision-based automation over screenshots. Our annotations encompass full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. We evaluate the dataset's efficacy using both open-weight and proprietary models in zero-shot, fine-tuned, and transfer learning settings. We further demonstrate positive transfer to web applications, highlighting its potential beyond mobile applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
