超越通过或失败：面向基于目标的移动UI导航的基础模型多维度基准测试
Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation
AI总结：
针对现有基准用二元指标评估基于基础模型的移动UI导航存在的局限，提出综合基准Sphinx，通过多维度评估揭示FMs的失败模式，为改进其UI导航能力提供方向。
摘要：
基础模型（FMs）的最新进展使得基于高层目标指令导航移动应用程序（apps）成为可能，在UI测试等领域具有重要的工业应用价值。现有基准测试使用二元通过/失败指标评估基于FM的UI导航，但存在两大局限：无法反映移动UI导航的复杂性——FMs可能因多种原因（如误解指令、规划失败）失败；任务过于简化，难以代表真实场景，缺乏工业相关性。为解决这些局限，我们提出Sphinx，一个在工业UI导航场景下对FMs进行多维度评估的综合基准。Sphinx引入了一个专用工具包，评估FMs的五项关键能力，提供对应用知识不足或规划问题等失败模式的详细洞察。我们使用流行的Google Play应用和微信内部UI测试用例，评估了8个FMs的20种不同配置。结果显示，现有FMs在基于目标的测试任务中普遍表现不佳，主要原因是UI特定能力不足。我们总结了通过Sphinx基准测试FMs得到的七条经验，为改进基于FM的移动UI导航提供了明确方向。
英文摘要：
Recent advances of foundation models (FMs) have made navigating mobile applications (apps) based on high-level goal instructions within reach, with significant industrial applications such as UI testing. While existing benchmarks evaluate FM-based UI navigation using the binary pass/fail metric, they have two major limitations: they cannot reflect the complex nature of mobile UI navigation where FMs may fail for various reasons (e.g., misunderstanding instructions and failed planning), and they lack industrial relevance due to oversimplified tasks that poorly represent real-world scenarios. To address the preceding limitations, we propose Sphinx, a comprehensive benchmark for multi-dimensional evaluation of FMs in industrial settings of UI navigation. Sphinx introduces a specialized toolkit that evaluates five essential FM capabilities, providing detailed insights into failure modes such as insufficient app knowledge or planning issues. Using both popular Google Play applications and WeChat's internal UI test cases, we evaluate 8 FMs with 20 different configurations. Our results show that existing FMs universally struggle with goal-based testing tasks, primarily due to insufficient UI-specific capabilities. We summarize seven lessons learned from benchmarking FMs with Sphinx, providing clear directions for improving FM-based mobile UI navigation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
