Pre-Flight: 评估大型语言模型航空运营知识的基准
Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge
AI总结：
提出Pre-Flight基准，包含300道多选题，评估LLM在航空运营知识上的表现，发现最强模型准确率82.7%，低于专家水平的95%，表明领域特定评估的必要性。
摘要：
大型语言模型（LLM）越来越多地被提议用于航空业务运营，从文档和培训生成到面向客户的助手。通用基准无法衡量模型是否安全且正确地推理航空特定运营知识，而该领域的高风险、受监管性质使得这一差距至关重要。我们提出Pre-Flight，一个包含300道多选题的开源基准，题目来自国际标准和机场地面运营材料，涵盖国际机场地面运营、ICAO和美国FAA法规、航空通用知识和复杂运营场景。题目由具有空中交通管理、地面运营和商业飞行经验的从业者编写和审核。我们使用Inspect评估框架评估了一系列当代商业和开放权重模型，根据标准多选题协议下的准确率进行评分，并随着新模型的发布滚动维护排行榜。与在会议上对航空专业人士进行的小样本测验中获得的约95%的非正式专家参考值相比，即使是最强的评估模型（2026年发布）也仅达到82.7%，从2025年初的约75%逐步提高。因此，仍然存在一个显著且持久的低于专家级可靠性的差距。我们发布了数据集、评估工具和结果，该基准可在inspect_evals分发的社区评估包中获得。我们认为，这种领域特定评估是在非安全关键航空运营中负责任地部署生成式AI的必要前提。
英文摘要：
Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge, and the high stakes, regulated nature of the domain makes that gap consequential. We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%, having improved only gradually from roughly 75% in early 2025. A substantial and persistent gap below expert level reliability therefore remains. We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. We argue that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
