面向知识密集型问答任务的自动数据集生成
Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks
AI总结：
针对现有QA系统及RAG在复杂推理上的不足，提出通过LLM自动生成基于上下文的问答对来微调模型的方法，实验证明该方法可提升模型逻辑与事实准确性，Mistral-7b-v0.3表现优于Llama-3-8b。
摘要：
问答（QA）系统旨在知识库中检索合适的答案。当前的QA系统在处理需要复杂推理或实时知识整合的查询时存在困难，通常会辅以针对数据源的检索技术，例如检索增强生成（Retrieval-Augmented Generation, RAG）。但RAG在处理复杂推理以及多源信息间的逻辑关联方面仍面临挑战。本文提出一种提升大语言模型（Large Language Models, LLMs）在知识密集型QA任务中表现的新方法，核心是自动生成基于上下文的问答对。该方法利用LLMs生成微调数据，减少对人工标注的依赖，同时提升模型的理解与推理能力。所提出的系统包含自动问答生成器和模型微调模块，采用困惑度、ROUGE、BLEU和BERTScore进行评估。全面实验表明，该方法在逻辑连贯性和事实准确性上均有提升，对开发适应性强的人工智能（AI）系统具有参考意义。在LLM生成的问答对测试中，Mistral-7b-v0.3的表现优于Llama-3-8b，其BERT F1、BLEU和ROUGE得分分别为0.858、0.172和0.260，而人工标注问答对对应的得分为0.836、0.083和0.139。
英文摘要：
A question-answering (QA) system is to search suitable answers within a knowledge base. Current QA systems struggle with queries requiring complex reasoning or real-time knowledge integration. They are often supplemented with retrieval techniques on a data source such as Retrieval-Augmented Generation (RAG). However, RAG continues to face challenges in handling complex reasoning and logical connections between multiple sources of information. A novel approach for enhancing Large Language Models (LLMs) in knowledge-intensive QA tasks is presented through the automated generation of context-based QA pairs. This methodology leverages LLMs to create fine-tuning data, reducing reliance on human labelling and improving model comprehension and reasoning capabilities. The proposed system includes an automated QA generator and a model fine-tuner, evaluated using perplexity, ROUGE, BLEU, and BERTScore. Comprehensive experiments demonstrate improvements in logical coherence and factual accuracy, with implications for developing adaptable Artificial Intelligence (AI) systems. Mistral-7b-v0.3 outperforms Llama-3-8b with BERT F1, BLEU, and ROUGE scores 0.858, 0.172, and 0.260 of for the LLM generated QA pairs compared to scores of 0.836, 0.083, and 0.139 for the human annotated QA pairs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
