STELLA：面向航空航天信息检索基准的自反思术语感知框架
STELLA: Self-Reflective Terminology-Aware Framework for Building an Aerospace Information Retrieval Benchmark
AI总结：
STELLA框架通过术语感知和自反思方法构建航空航天信息检索基准，评估嵌入模型的词汇和语义匹配能力。
摘要：
航空航天行业中的任务严重依赖于搜索和重用大量技术文档，但目前尚没有一个公开的信息检索（IR）基准能够反映该领域的术语性和查询意图特征。为了解决这一差距，本文提出了STELLA（Self-Reflective TErminoLogy-Aware Framework for BuiLding an Aerospace Information Retrieval Benchmark）框架。利用该框架，我们引入了STELLA基准，这是一个基于NASA技术报告服务器（NTRS）文档构建的航空航天专用信息检索评估集，通过包含文档布局检测、段落分块、术语词典构建、合成查询生成和跨语言扩展的系统流程构建而成。该框架生成两种类型的查询：术语一致查询（TCQ），包含术语原文以评估词汇匹配，以及术语无关查询（TAQ），利用术语的描述来评估语义匹配。这使得能够对嵌入模型的词汇和语义匹配能力进行分离评估。此外，我们结合链式密度（CoD）和自反思方法与查询生成相结合，以提高质量并实现一种反映真实用户查询实践的混合跨语言扩展。在STELLA基准上对七种嵌入模型的评估显示，大型解码器基嵌入模型在语义理解方面表现最强，而在需要精确词汇匹配技术术语的领域中，词汇匹配方法如BM25仍然具有很强的竞争力。STELLA基准为可靠评估和改进嵌入模型在航空航天领域信息检索任务中的性能提供了可重复的基础。STELLA基准可在https://huggingface.co/datasets/telepix/STELLA上找到。
英文摘要：
Tasks in the aerospace industry heavily rely on searching and reusing large volumes of technical documents, yet there is no public information retrieval (IR) benchmark that reflects the terminology- and query-intent characteristics of this domain. To address this gap, this paper proposes the STELLA (Self-Reflective TErminoLogy-Aware Framework for BuiLding an Aerospace Information Retrieval Benchmark) framework. Using this framework, we introduce the STELLA benchmark, an aerospace-specific IR evaluation set constructed from NASA Technical Reports Server (NTRS) documents via a systematic pipeline that comprises document layout detection, passage chunking, terminology dictionary construction, synthetic query generation, and cross-lingual extension. The framework generates two types of queries: the Terminology Concordant Query (TCQ), which includes the terminology verbatim to evaluate lexical matching, and the Terminology Agnostic Query (TAQ), which utilizes the terminology's description to assess semantic matching. This enables a disentangled evaluation of the lexical and semantic matching capabilities of embedding models. In addition, we combine Chain-of-Density (CoD) and the Self-Reflection method with query generation to improve quality and implement a hybrid cross-lingual extension that reflects real user querying practices. Evaluation of seven embedding models on the STELLA benchmark shows that large decoder-based embedding models exhibit the strongest semantic understanding, while lexical matching methods such as BM25 remain highly competitive in domains where exact lexical matching technical term is crucial. The STELLA benchmark provides a reproducible foundation for reliable performance evaluation and improvement of embedding models in aerospace-domain IR tasks. The STELLA benchmark can be found in https://huggingface.co/datasets/telepix/STELLA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
