DIRAS：检索增强生成中文档相关性的高效LLM标注
DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation
AI总结：
针对RAG中领域特定查询的相关性标注挑战，提出DIRAS方案，通过微调开源LLM实现无需人工标注的细致相关性标注与评分，使8B模型达到GPT-4级性能，助力实际RAG开发。
摘要：
检索增强生成（RAG）被广泛用于将查询的响应基于特定领域的文档。但是，当回答需要信息综合分析（例如，“告诉我今天股市的好消息”）的查询时，RAG实现是否会遗漏重要信息？为了解决这些问题，RAG开发者需要为其感兴趣的领域标注信息检索（IR）数据，这具有挑战性，因为（1）领域特定的查询通常需要超越浅层语义相关性的细致相关性定义；（2）人工或GPT-4标注成本高昂，且无法覆盖所有（查询，文档）对（即标注选择偏差），从而损害了评估IR召回率的有效性。为应对这些挑战，我们提出了DIRAS（具有可扩展性的领域特定信息检索标注），这是一种无需人工标注的方案，它微调开源LLM以考虑细致相关性定义，并使用校准的相关性分数标注（部分）相关性标签。广泛评估表明，DIRAS使较小的（8B）LLM在标注和排序未见（查询，文档）对上达到GPT-4级性能，并对实际RAG开发有帮助。所有代码、LLM生成和人工标注可在https://github.com/EdisonNi-hku/DIRAS中找到。
英文摘要：
Retrieval Augmented Generation (RAG) is widely employed to ground responses to queries on domain-specific documents. But do RAG implementations leave out important information when answering queries that need an integrated analysis of information (e.g., Tell me good news in the stock market today.)? To address these concerns, RAG developers need to annotate information retrieval (IR) data for their domain of interest, which is challenging because (1) domain-specific queries usually need nuanced definitions of relevance beyond shallow semantic relevance; and (2) human or GPT-4 annotation is costly and cannot cover all (query, document) pairs (i.e., annotation selection bias), thus harming the effectiveness in evaluating IR recall. To address these challenges, we propose DIRAS (Domain-specific Information Retrieval Annotation with Scalability), a manual-annotation-free schema that fine-tunes open-sourced LLMs to consider nuanced relevance definition and annotate (partial) relevance labels with calibrated relevance scores. Extensive evaluation shows that DIRAS enables smaller (8B) LLMs to achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs, and is helpful for real-world RAG development. All code, LLM generations, and human annotations can be found in \url{https://github.com/EdisonNi-hku/DIRAS}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
