KGCE：基于多模态语言模型的跨平台教育代理基准评估知识增强双图评估器
KGCE: Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models
AI总结：
KGCE提出一种基于多模态语言模型的跨平台教育代理基准评估框架，通过知识库增强和双图评估方法提升对特定学校软件任务的执行效率和评估精度。
摘要：
随着多模态大语言模型（MLMs）在自主代理中的快速应用，教育环境中跨平台任务执行能力受到了广泛关注。然而，现有基准框架在支持教育情境中的跨平台任务方面仍然存在显著不足，尤其是在处理特定学校软件（如小雅智能助手、华世小字等）时，由于缺乏对这些私有领域软件结构细节的理解，代理的效率往往会大幅下降。此外，当前的评估方法严重依赖于粗粒度的指标，如目标导向或轨迹匹配，这使得在复杂任务中难以捕捉代理的执行细节和效率。为了解决这些问题，我们提出了KGCE（基于多模态语言模型的跨平台教育代理基准评估知识增强双图评估器），一种新的基准平台，集成了知识库增强和双图评估框架。我们首先构建了一个包含104个教育相关任务的数据集，涵盖Windows、Android和跨平台协作任务。KGCE引入了一个双图评估框架，将任务分解为多个子目标并验证其完成状态，提供细粒度的评估指标。为了克服现有代理在私有领域任务中的执行瓶颈，我们开发了一个增强的代理系统，集成了针对特定学校软件的知识库。代码可在https://github.com/Kinginlife/KGCE中找到。
英文摘要：
With the rapid adoption of multimodal large language models (MLMs) in autonomous agents, cross-platform task execution capabilities in educational settings have garnered significant attention. However, existing benchmark frameworks still exhibit notable deficiencies in supporting cross-platform tasks in educational contexts, especially when dealing with school-specific software (such as XiaoYa Intelligent Assistant, HuaShi XiaZi, etc.), where the efficiency of agents often significantly decreases due to a lack of understanding of the structural specifics of these private-domain software. Additionally, current evaluation methods heavily rely on coarse-grained metrics like goal orientation or trajectory matching, making it challenging to capture the detailed execution and efficiency of agents in complex tasks. To address these issues, we propose KGCE (Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models), a novel benchmarking platform that integrates knowledge base enhancement and a dual-graph evaluation framework. We first constructed a dataset comprising 104 education-related tasks, covering Windows, Android, and cross-platform collaborative tasks. KGCE introduces a dual-graph evaluation framework that decomposes tasks into multiple sub-goals and verifies their completion status, providing fine-grained evaluation metrics. To overcome the execution bottlenecks of existing agents in private-domain tasks, we developed an enhanced agent system incorporating a knowledge base specific to school-specific software. The code can be found at https://github.com/Kinginlife/KGCE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
