AeroCopilotBench：用于在交互式虚拟驾驶舱环境中评估作为航空副驾驶的大语言模型智能体的双层基准

中文摘要 AI 辅助

大语言模型（LLM）智能体可协助飞行机组完成复杂决策与任务执行，但现有航空评估以静态知识为核心，无法对交互式环境中的流程执行与安全合规性开展系统性测试。本文提出AeroCopilot操作环境（ACOE）——可复现的交互式虚拟驾驶舱测试环境，以及AeroCopilotBench——双层航空智能体评估基准。第一层（Tier-1）通过1200道多项选择题评估航空知识，第二层（Tier-2）包含73项由制造商《飞行员操作手册》（POHs）衍生并在ACOE中实例化的紧急与异常任务。ACOE将自然语言流程转换为可执行的状态转换、最终状态目标条件及硬性安全约束，使模型能解读驾驶舱状态、诊断故障并通过标准化工具接口操作飞机系统。我们建立安全门控评估框架，轨迹仅在达成所有任务目标且未违反任何硬性安全约束时才算成功，同时分别衡量安全目标进度与轨迹安全性。在12个模型中，第二层最高成功率为72.6%，而静态知识表现无法一致转化为流程执行能力。对3个代表性模型的451个失败案例分析发现，存在流程完整性、状态反馈使用及长时序执行管理方面的反复失败。这些发现推动了感知状态的智能体编排、任务完成与轨迹安全的联合评估及重复回归测试。ACOE与AeroCopilotBench为测试航空智能体的知识应用、交互式执行及操作安全性提供了可复现的基础。

英文摘要

Aviation knowledge question answering cannot directly assess the operational effectiveness and safety compliance of large language models throughout aircraft emergency procedures. We introduce AeroCopilotBench and its executable cockpit environment, ACOE, which define state-transition rules, task goals, and trajectory-level safety constraints based on aircraft-specific Pilot's Operating Handbooks (POHs). The benchmark comprises 12 scenario templates and 73 tasks across two aircraft types, evaluating task completion, safety compliance, execution discipline, and repeatability. Across repeated evaluations of 12 models, the highest safety-gated success rate is 72.6\%. Most failed episodes achieve all critical goals but do not satisfy all remaining terminal goals. Across repeated runs, models still omit steps that they execute in other runs of the same task. Analysis of failed trajectories further reveals that some reasons for actions that conflict with the aircraft's POH recur across models. These results expose shortcomings in complete procedure execution, consistency across runs, and aircraft-specific emergency response.

发表机构

• School of Aeronautic Science and Engineering, Beihang University（北京航空航天大学航空科学与工程学院）

机构由 AI 辅助整理，请以论文原文为准。