面向航空领域的句向量Transformer适配方法
Adapting Sentence Transformers for the Aviation Domain
AI总结：
本文提出两阶段方法适配航空领域句向量Transformer，先以TSDAE进行航空文本预训练，再在SBERT架构中用NLI数据微调，显著提升航空下游任务性能。
摘要：
学习有效的句子表示对于许多自然语言处理（NLP）任务至关重要，包括语义搜索、语义文本相似度（STS）和聚类。虽然已有多种Transformer模型被开发用于句子嵌入学习，但这些模型在处理航空等专业领域时可能无法达到最优性能，因为航空领域具有技术术语、缩写和非传统语法等独特特征。此外，标注数据集的缺乏使得专门为航空领域训练模型变得困难。为了解决这些挑战，我们提出了一种面向航空领域适配句向量Transformer的新方法。我们的方法是一个两阶段过程，包括预训练和微调。在预训练阶段，我们使用Transformer和序列去噪自编码器（TSDAE），以航空文本数据作为输入来提升初始模型性能。随后，我们在句向量双向编码器表示（SBERT）架构中使用自然语言推理（NLI）数据集对模型进行微调，以缓解过拟合问题。在多个下游任务上的实验结果表明，我们适配后的句向量Transformer显著优于通用Transformer，证明了我们的方法在捕捉航空领域细微差异方面的有效性。总体而言，我们的工作突显了领域特定适配在为航空等专业行业开发高质量NLP解决方案中的重要性。
英文摘要：
Learning effective sentence representations is crucial for many Natural Language Processing (NLP) tasks, including semantic search, semantic textual similarity (STS), and clustering. While multiple transformer models have been developed for sentence embedding learning, these models may not perform optimally when dealing with specialized domains like aviation, which has unique characteristics such as technical jargon, abbreviations, and unconventional grammar. Furthermore, the absence of labeled datasets makes it difficult to train models specifically for the aviation domain. To address these challenges, we propose a novel approach for adapting sentence transformers for the aviation domain. Our method is a two-stage process consisting of pre-training followed by fine-tuning. During pre-training, we use Transformers and Sequential Denoising AutoEncoder (TSDAE) with aviation text data as input to improve the initial model performance. Subsequently, we fine-tune our models using a Natural Language Inference (NLI) dataset in the Sentence Bidirectional Encoder Representations from Transformers (SBERT) architecture to mitigate overfitting issues. Experimental results on several downstream tasks show that our adapted sentence transformers significantly outperform general-purpose transformers, demonstrating the effectiveness of our approach in capturing the nuances of the aviation domain. Overall, our work highlights the importance of domain-specific adaptation in developing high-quality NLP solutions for specialized industries like aviation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
