应用开发过程中利用大语言模型为UI图标推断替代文本(alt-text)
Inferring Alt-text For UI Icons With Large Language Models During App Development
AI总结:
针对移动UI图标缺alt-text影响无障碍性、现有方法存在数据需求高或不适用开发迭代期的问题,研究提出用含多维度上下文的部分UI数据微调LLM得到IconDesc,在小数据集训练下生成alt-text效果显著提升,助力UI无障碍开发迭代。
摘要:
确保移动应用的无障碍性仍是一项重大挑战,对依赖屏幕阅读器的视障用户而言尤其如此。用户界面(UI)图标是导航与交互的核心要素,却往往缺乏有意义的替代文本(alt-text),为用户的有效使用造成障碍。传统的深度学习alt-text生成方法需要海量数据集,且难以应对图标类型的多样性与不均衡问题。近年出现的视觉语言模型(VLM)需要完整的UI屏幕信息,这在应用开发的迭代阶段并不现实。为解决这些问题,我们提出一种利用大语言模型(LLM)的新方法,可基于部分UI数据自主为移动UI图标生成信息丰富的alt-text。通过纳入图标上下文(包括类别、资源ID、边界、OCR检测文本,以及来自父节点和兄弟节点的上下文信息),我们在约1400个图标的小型数据集上微调了一款现成LLM,得到IconDesc模型。在实证评估与用户研究中,IconDesc在生成相关alt-text方面表现出显著提升。这一能力使IconDesc成为开发者的宝贵工具,可助力快速迭代并提升UI无障碍性。
英文摘要:
Ensuring accessibility in mobile applications remains a significant challenge, particularly for visually impaired users who rely on screen readers. User interface icons are essential for navigation and interaction and often lack meaningful alt-text, creating barriers to effective use. Traditional deep learning approaches for generating alt-text require extensive datasets and struggle with the diversity and imbalance of icon types. More recent Vision Language Models (VLMs) require complete UI screens, which can be impractical during the iterative phases of app development. To address these issues, we introduce a novel method using Large Language Models (LLMs) to autonomously generate informative alt-text for mobile UI icons with partial UI data. By incorporating icon context, that include class, resource ID, bounds, OCR-detected text, and contextual information from parent and sibling nodes, we fine-tune an off-the-shelf LLM on a small dataset of approximately 1.4k icons, yielding IconDesc. In an empirical evaluation and a user study IconDesc demonstrates significant improvements in generating relevant alt-text. This ability makes IconDesc an invaluable tool for developers, aiding in the rapid iteration and enhancement of UI accessibility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。