发表机构
Jiutian Research; China Mobile（中移九天; 中国移动）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出CRATE（含安全评估扩展版CRATE-S）这一VLM-as-judge框架，通过步骤级推理解决移动智能体轨迹评估的上下文过载与安全缺失问题，在AndroidWorld、MobileRisk数据集上取得优于现有方法的性能。
AI 中文摘要
近期，对语言引导的移动智能体的评估已从基于规则的方法转向基于模型的方法，以实现可扩展的自动化评估。然而，现有的整体评估范式一次性处理完整轨迹，导致严重的上下文过载问题，且它们主要关注任务完成情况，却忽略了操作安全性。为解决这些局限，我们提出CRATE，一种新型的两阶段VLM-as-judge框架，用于移动智能体的自动评估，该框架兼容开源和闭源模型。CRATE利用步骤级结果推理机制，在每一步独立提取与任务相关的视觉线索，并推断动作条件下的状态变化，随后将得到的步骤级文本证据通过轨迹级聚合进行综合，以提供基于证据的任务完成评估。在此评估方案基础上，我们进一步将CRATE扩展为用于操作安全性评估的CRATE-S。大量实验验证了CRATE和CRATE-S的有效性与鲁棒性：CRATE基于Qwen2.5-VL-72B-Instruct在AndroidWorld上达到0.833的F1分数，优于SPA-Bench 20%；CRATE-S在MobileRisk上达到0.697的F1分数，展现出与基准真实值的强一致性。代码可在该https URL获取。
英文摘要
Evaluating language-guided mobile agents has recently shifted from rule-based to model-based approaches to achieve scalable and automated assessments. However, existing holistic evaluation paradigms process entire trajectories at once, leading to substantial context overload. Moreover, they primarily focus on task completion while overlooking operational safety. To address these limitations, we introduce CRATE, a novel two-stage VLM-as-judge framework for automated mobile agent evaluation that is compatible with both open- and closed-source models. Leveraging a step-level consequence reasoning mechanism, CRATE independently extracts task-relevant visual clues and infers action-conditioned state changes at each step. The resulting step-level textual evidence is then synthesized through trajectory-level aggregation to deliver an evidence-grounded evaluation of task completion. Building upon this evaluation scheme, we further extend CRATE to CRATE-S for operational safety assessment. Extensive experiments validate the effectiveness and robustness of both CRATE and CRATE-S. Powered by Qwen2.5-VL-72B-Instruct, CRATE achieves an F1-score of 0.833 on AndroidWorld (outperforming SPA-Bench by 20%), while CRATE-S reaches an F1-score of 0.697 on MobileRisk, demonstrating strong alignment with benchmark ground truths. Code is available at https://anonymous.4open.science/r/CRATE-D580.
