CAMB：面向民航维修的综合性工业级大语言模型基准
CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance
AI总结：
本文提出工业级民航维修 LLM 基准 CAMB，用于评估领域知识与复杂推理，并测试主流向量嵌入模型和 LLM 的 RAG 表现，且已开源。
摘要：
民航维修是一个以严格行业标准为特征的领域。在该领域中，维修程序和故障排除属于关键且知识密集型任务，需要复杂的推理能力。为弥补大语言模型（LLM）在这一垂直领域缺乏专门评估工具的问题，我们提出并构建了一个专为民航维修设计的工业级基准。该基准具有双重用途：它提供了一个标准化工具，用于衡量 LLM 在民航维修中的能力，并识别其在领域知识和复杂推理方面的具体差距。通过准确定位这些不足，该基准为有针对性的改进工作（例如面向领域的微调、RAG 优化或专门的提示工程）奠定基础，最终推动民航维修向更智能的解决方案发展。我们的工作弥补了当前 LLM 评估中的一个显著空白，即现有评估主要集中于数学和编程推理任务。此外，鉴于检索增强生成（RAG）系统目前是实际应用中的主流解决方案，我们利用该基准评估了现有的知名向量嵌入模型和 LLM 在民航维修场景中的表现。通过实验探索与分析，我们证明了该基准在评估模型于该领域内性能方面的有效性，并开源了这一评估基准和代码，以促进进一步的研究与开发：https://github.com/CamBenchmark/cambenchmark
英文摘要：
Civil aviation maintenance is a domain characterized by stringent industry standards. Within this field, maintenance procedures and troubleshooting represent critical, knowledge-intensive tasks that require sophisticated reasoning. To address the lack of specialized evaluation tools for large language models (LLMs) in this vertical, we propose and develop an industrial-grade benchmark specifically designed for civil aviation maintenance. This benchmark serves a dual purpose: It provides a standardized tool to measure LLM capabilities within civil aviation maintenance, identifying specific gaps in domain knowledge and complex reasoning. By pinpointing these deficiencies, the benchmark establishes a foundation for targeted improvement efforts (e.g., domain-specific fine-tuning, RAG optimization, or specialized prompt engineering), ultimately facilitating progress toward more intelligent solutions within civil aviation maintenance. Our work addresses a significant gap in the current LLM evaluation, which primarily focuses on mathematical and coding reasoning tasks. In addition, given that Retrieval-Augmented Generation (RAG) systems are currently the dominant solutions in practical applications , we leverage this benchmark to evaluate existing well-known vector embedding models and LLMs for civil aviation maintenance scenarios. Through experimental exploration and analysis, we demonstrate the effectiveness of our benchmark in assessing model performance within this domain, and we open-source this evaluation benchmark and code to foster further research and development:https://github.com/CamBenchmark/cambenchmark
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
