arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.08088cs.CL
收藏

通过LLM蒸馏适应金融文件的嵌入模型

Adaptation of Embedding Models to Financial Filings via LLM Distillation

Eliot Brenner, Dominic Seyler, Manjunath Hegde, Andrei Simion, Koustuv Dasgupta, Bing Xiang

AI总结:

本文通过LLM蒸馏方法,提升金融领域检索性能,实现MRR@5和DCG@5的显著提升。

摘要:

尽管生成式大语言模型(LLMs)取得了进展,但专用对话AI代理的实用应用仍受到计算成本、延迟要求和精确领域特定相关性测量的需求的限制。现有的嵌入模型解决了前两个限制,但在金融等专业领域的信息检索中表现不佳。本文介绍了一种可扩展的管道,该管道通过使用通用检索嵌入模型作为基础,从无标签语料库中训练专用模型。我们的方法在14种金融文件类型上,针对21,800个查询-文档对,平均提高了MRR@5 27.7%,平均提高了mean DCG@5 44.6%,并在FinanceBench的3种文档类别中提高了NDCG。我们通过LLM判断的相关性来适应检索嵌入(双编码器)用于RAG,而不是LLM生成器,将领域知识蒸馏成一个紧凑的检索器。有先前的工作将合成生成的查询与真实段落配对,直接微调检索模型。我们的管道不同之处在于引入了学生模型和教师模型之间的交互,将基于检索的挖掘硬正/负示例与迭代重新训练学生模型的权重相结合。每次检索迭代都使用改进的学生模型从语料库中挖掘逐步更难的训练示例,用于后续的训练迭代。该方法提供了一种成本效益高的解决方案,以弥合通用模型与专业领域之间的差距,而无需劳工密集型的人类注释。

英文摘要:

Despite advances in generative large language models (LLMs), practical application of specialized conversational AI agents remains constrained by computation costs, latency requirements, and the need for precise domain-specific relevance measures. While existing embedding models address the first two constraints, they underperform on information retrieval in specialized domains like finance. This paper introduces a scalable pipeline that trains specialized models from an unlabeled corpus using a general purpose retrieval embedding model as foundation. Our method yields an average of 27.7% improvement in MRR$\texttt{@}$5, 44.6% improvement in mean DCG$\texttt{@}$5 across 14 financial filing types measured over 21,800 query-document pairs, and improved NDCG on 3 of 4 document classes in FinanceBench. We adapt retrieval embeddings (bi-encoder) for RAG, not LLM generators, using LLM-judged relevance to distill domain knowledge into a compact retriever. There are prior works which pair synthetically generated queries with real passages to directly fine-tune the retrieval model. Our pipeline differs from these by introducing interaction between student and teacher models that interleaves retrieval-based mining of hard positive/negative examples from the unlabeled corpus with iterative retraining of the student model's weights using these examples. Each retrieval iteration uses the refined student model to mine the corpus for progressively harder training examples for the subsequent training iteration. The methodology provides a cost-effective solution to bridging the gap between general-purpose models and specialized domains without requiring labor-intensive human annotation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑