arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.09342cs.HC
收藏

多模态UI设计与开发工具中草图符号的可供性

Affordances of Sketched Notations for Multimodal UI Design and Development Tools

Sam H. Ross, Yunseo Lee, Coco K. Lee, Jayne Everson, R. Benjamin Shapiro

AI总结:

本研究使用认知维度框架分析两种UI草图符号体系,发现灵活的FlexiSketch符号比固定的FixedSketch符号更能支持直观创作,但需采用基于transformer和人机协同的强化学习等现代AI方法才能实现。

摘要:

解释UI草图或自然语言描述的多模态UI设计与开发工具本质上具有符号体系:即它们能理解的输入。在基于AI的系统中,符号体系由用于训练这些系统的数据隐式定义。为了为交互式设计系统创建可用且直观的符号体系,我们必须将训练数据集视为符号规范来进行审视、设计和评估。为了更好地理解未来设计工具的符号可能性设计空间,我们使用符号的认知维度框架来分析UI草图绘制的两种可能符号体系。第一种符号体系是现有UI草图数据集的草图绘制规则,第二种符号体系是本研究中参与者生成的草图集合,其中个人在不受强制表征规则限制的情况下绘制UI。我们设想了分别用每种符号体系构建的两个系统,FixedSketch和FlexiSketch,以理解系统的差异性可供性及潜在设计需求。我们发现,参与者的草图由元素级符号组成,这些符号孤立时存在歧义,但在整个设计的上下文中是可解释的。在许多认知维度上,FlexiSketch符号体系比FixedSketch符号体系支持更直观的创造性表达,且所需认知努力更低,但无法通过当前基于元素的UI草图识别方法来支持。我们认为,未来的多模态设计工具若要真正做到以人为本,必须采用现代AI方法,包括基于transformer和人机协同的强化学习技术,以理解用户富含上下文的表达性符号及修正。

英文摘要:

Multimodal UI design and development tools that interpret sketches or natural language descriptions of UIs inherently have notations: the inputs they can understand. In AI-based systems, notations are implicitly defined by the data used to train these systems. In order to create usable and intuitive notations for interactive design systems, we must regard, design, and evaluate these training datasets as notation specifications. To better understand the design space of notational possibilities for future design tools, we use the Cognitive Dimensions of Notations framework to analyze two possible notations for UI sketching. The first notation is the sketching rules for an existing UI sketch dataset, and the second notation is the set of sketches generated by participants in this study, where individuals sketched UIs without imposed representational rules. We imagine two systems, FixedSketch and FlexiSketch, built with each notation respectively, in order to understand the differential affordances of, and potential design requirements for, systems. We find that participants' sketches were composed of element-level notations that are ambiguous in isolation but are interpretable in context within whole designs. For many cognitive dimensions, the FlexiSketch notation supports greater intuitive creative expression and affords lower cognitive effort than the FixedSketch notation, but cannot be supported with prevailing, element-based approaches to UI sketch recognition. We argue that for future multimodal design tools to be truly human-centered, they must adopt contemporary AI methods, including transformer-based and human-in-the-loop, reinforcement learning techniques to understand users' context-rich expressive notations and corrections.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑