抑制构造LLM中的领域特定幻觉:图RAG和QLoRA在河流和泥沙控制技术标准上的知识图谱基础
Suppressing Domain-Specific Hallucination in Construction LLMs: A Knowledge Graph Foundation for GraphRAG and QLoRA on River and Sediment Control Technical Standards
AI总结:
本文通过知识图谱和QLoRA技术,解决基于日本河流和泥沙控制技术标准的问答问题,发现领域特定微调在质量和效率上优于GraphRAG。
摘要:
本文针对使用开源大语言模型回答源自日本《河流和泥沙控制技术标准》(多卷法规文件)的技术问题挑战,采用三种互补方法:Case A(普通20B LLM基线)、Case B(8B LLM在715个图导出QA对上使用QLoRA领域微调)、Case C(20B LLM通过GraphRAG集成Neo4j知识图谱)。所有案例均使用日本定制的Swallow系列LLM,并在涵盖8个技术类别的100个问题基准上进行评估,由独立LLM(Qwen2.5-14B)自动评分(0-3)。关键发现是性能反转:8B QLoRA微调模型(Case B)获得2.92/3的评分,超越20B基线(Case A:2.29/3,+0.63)和20B GraphRAG方法(Case C:2.62/3,+0.30),且运行延迟快3倍(14.2秒 vs. 42.2秒)。GraphRAG提供适度提升(+0.33 over baseline),但两者在质量和效率上均被领域特定微调所超越。我们记录了完整的工程流程,包括知识图谱构建(200个节点,268个关系)、QLoRA训练数据从Neo4j关系生成、单GPU(16GB VRAM)使用unsloth、GGUF Q4_K_M量化和Ollama部署,以及图检索和重排序设计。高层工程经验在正文汇总;实施陷阱和工具链细节记录在补充材料中。
英文摘要:
This paper addresses the challenge of answering technical questions derived from Japan's River and Sediment Control Technical Standards -- a multi-volume regulatory document covering survey, planning, design, and maintenance of river levees, dams, and sabo structures -- using open-source large language models running entirely on local hardware. We implement and evaluate three complementary approaches: Case A (plain 20B LLM baseline), Case B (8B LLM with QLoRA domain fine-tuning on 715 graph-derived QA pairs), and Case C (20B LLM augmented with a Neo4j knowledge graph via GraphRAG). All three cases use the Swallow series of Japanese-adapted LLMs and are evaluated on a 100-question benchmark spanning 8 technical categories, judged automatically by an independent LLM (Qwen2.5-14B, score 0--3). The key finding is a performance inversion: the 8B QLoRA fine-tuned model (Case B) achieves a judge average of 2.92/3 -- surpassing both the 20B plain baseline (Case A: 2.29/3, $+$0.63) and the 20B GraphRAG approach (Case C: 2.62/3, $+$0.30) -- while running at 3$\times$ faster latency (14.2s vs. 42.2s for Case A). GraphRAG provides moderate gains ($+$0.33 over baseline) but is outperformed by domain-specific fine-tuning in both quality and efficiency. We document the full engineering pipeline, including knowledge graph construction (200 nodes, 268 relations), QLoRA training data generation from Neo4j relations, training on a single GPU (16 GB VRAM) using unsloth, GGUF Q4_K_M quantisation and Ollama deployment, and the graph retrieval and re-ranking design. High-level engineering lessons are distilled in the main body; implementation pitfalls and toolchain details are documented in Supplementary Materials.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。