PilotBench：一个具有安全约束的一般航空智能体基准
PilotBench: A Benchmark for General Aviation Agents with Safety Constraints
AI总结：
PilotBench通过对比LLM与传统预报器，评估飞行轨迹和姿态预测，揭示了传统方法在精度上的优势与LLM在指令遵循上的优势，指出LLM在高负荷阶段表现下降，推动混合架构的发展。
摘要：
随着大语言模型（LLM）向具身AI代理发展，一个核心问题浮现：训练于文本语料的模型能否在物理环境中可靠推理复杂物理并遵守安全约束？我们通过PilotBench基准评估LLM在安全关键飞行轨迹和姿态预测上的表现。PilotBench基于708个真实世界的一般航空轨迹，涵盖九种不同的飞行阶段，同步34通道 telemetry 数据。通过对比分析LLM和传统预报器，我们引入Pilot-Score，该指标平衡60%的回归精度与40%的指令遵循和安全合规性。对41个模型的比较评估揭示了精度-可控性二元对立：传统预报器在MAE上达到7.01但缺乏语义推理能力，而LLM在指令遵循上达到86-89%但精度下降11-14%。相位分层分析进一步揭示了动态复杂性差距，LLM在高负荷阶段如爬升和进近时表现急剧下降，表明隐式物理模型脆弱。这些经验发现推动了结合LLM符号推理与专用预报器数值精度的混合架构发展。PilotBench为安全约束领域具身AI的进步提供了严谨的基础。
英文摘要：
As Large Language Models (LLMs) advance toward embodied AI agents operating in physical environments, a fundamental question emerges: can models trained on text corpora reliably reason about complex physics while adhering to safety constraints? We address this through PilotBench, a benchmark evaluating LLMs on safety-critical flight trajectory and attitude prediction. Built from 708 real-world general aviation trajectories spanning nine operationally distinct flight phases with synchronized 34-channel telemetry, PilotBench systematically probes the intersection of semantic understanding and physics-governed prediction through comparative analysis of LLMs and traditional forecasters. We introduce Pilot-Score, a composite metric balancing 60% regression accuracy with 40% instruction adherence and safety compliance. Comparative evaluation across 41 models uncovers a Precision-Controllability Dichotomy: traditional forecasters achieve superior MAE of 7.01 but lack semantic reasoning capabilities, while LLMs gain controllability with 86--89% instruction-following at the cost of 11--14 MAE precision. Phase-stratified analysis further exposes a Dynamic Complexity Gap-LLM performance degrades sharply in high-workload phases such as Climb and Approach, suggesting brittle implicit physics models. These empirical discoveries motivate hybrid architectures combining LLMs' symbolic reasoning with specialized forecasters' numerical precision. PilotBench provides a rigorous foundation for advancing embodied AI in safety-constrained domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
