arXiv 2503.024032025-03cs.AI

AutoEval：一种用于移动智能体自主评估的实用框架

AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents

Jiahui Sun, Zhichao Hua, Yubin Xia

AI总结：

AutoEval提出了一种无需人工的移动智能体评估框架，通过UI状态变化表示自动生成奖励信号并采用评判系统，实现高达94%的自主评估准确率。

摘要：

对移动智能体的全面评估可以显著推进其开发进程和实际应用性。然而，现有的基准测试因在定义任务奖励信号和实施评估代码方面需要大量人工努力而缺乏实用性和可扩展性。我们提出了AutoEval，一个无需任何人工努力即可测试移动智能体的评估框架。我们的方法设计了一种UI状态变化表示，可用于自动生成任务奖励信号，并采用了一个评判系统进行自主评估。评估表明，AutoEval能够自动生成与人工标注信号高度相关的奖励信号，并在自主评估中达到高达94%的高准确率，与人工评估相当。最后，我们使用我们的框架评估了最先进的移动智能体，提供了对其性能和局限性的见解。

英文摘要：

Comprehensive evaluation of mobile agents can significantly advance their development and real-world applicability. However, existing benchmarks lack practicality and scalability due to the extensive manual effort in defining task reward signals and implementing evaluation codes. We propose AutoEval, an evaluation framework which tests mobile agents without any manual effort. Our approach designs a UI state change representation which can be used to automatically generate task reward signals, and employs a Judge System for autonomous evaluation. Evaluation shows AutoEval can automatically generate reward signals with high correlation to human-annotated signals, and achieve high accuracy (up to 94%) in autonomous evaluation comparable to human evaluation. Finally, we evaluate state-of-the-art mobile agents using our framework, providing insights into their performance and limitations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文