arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.00634cs.SEcs.HC
收藏

生成式AI会使可用性测试过时吗?

Does GenAI Make Usability Testing Obsolete?

Universität Hamburg(汉堡大学)

Ali Ebrahimi Pourasad, Walid Maalej

AI总结:

本文提出UX-LLM,一种基于大视觉语言模型的可用性问题预测工具,虽无法完全替代传统测试,但可作为补充,尤其适用于资源有限的小型团队。

摘要:

确保可用性对于移动应用的成功至关重要。可用性问题可能会损害用户体验并影响对应用程序质量的感知。本文介绍了UX-LLM,一种由大型视觉语言模型驱动的新工具,用于预测iOS应用程序的可用性问题。为了评估UX-LLM的性能,我们预测了两个中等复杂度的开源应用程序的可用性问题,并请两位可用性专家评估预测结果。我们还对这两个应用程序进行了传统的可用性测试和专家审查,并将结果与UX-LLM的结果进行了比较。UX-LLM的精度范围从0.61到0.66,召回率在0.35到0.38之间,表明其能够识别有效的可用性问题,但未能捕捉到大多数问题。最后,我们与一个综合项目开发团队进行了焦点小组讨论,该团队正在开发一款为视障人士设计的交通应用程序。焦点小组对UX-LLM表达了积极的看法,因为它在他们的应用程序中识别了未知的可用性问题。然而,他们也提出了关于将其整合到开发工作流程中的担忧,这表明有改进的潜力。我们的结果表明,UX-LLM不能完全取代传统的可用性评估方法,但特别适用于资源有限的小型团队,以识别较少见的用户路径中的问题,因为它能够检查源代码。

英文摘要:

Ensuring usability is crucial for the success of mobile apps. Usability issues can compromise user experience and negatively impact the perceived app quality. This paper presents UX-LLM, a novel tool powered by a Large Vision-Language Model that predicts usability issues in iOS apps. To evaluate the performance of UX-LLM, we predicted usability issues in two open-source apps of a medium complexity and asked two usability experts to assess the predictions. We also performed traditional usability testing and expert review for both apps and compared the results to those of UX-LLM. UX-LLM demonstrated precision ranging from 0.61 and 0.66 and recall between 0.35 and 0.38, indicating its ability to identify valid usability issues, yet failing to capture the majority of issues. Finally, we conducted a focus group with an app development team of a capstone project developing a transit app for visually impaired persons. The focus group expressed positive perceptions of UX-LLM as it identified unknown usability issues in their app. However, they also raised concerns about its integration into the development workflow, suggesting potential improvements. Our results show that UX-LLM cannot fully replace traditional usability evaluation methods but serves as a valuable supplement particularly for small teams with limited resources, to identify issues in less common user paths, due to its ability to inspect the source code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑