评估 ChatGPT 在核领域特定数据上的表现
Evaluating ChatGPT on Nuclear Domain-Specific Data
AI总结：
本文评估 ChatGPT 在核领域问答中的表现，比较独立 LLM 与检索增强生成（RAG）方法，发现 RAG 能显著提升回答的准确性和上下文相关性。
摘要：
本文考察了大型语言模型（LLM）ChatGPT 在高度专业化的核数据领域中问答（Q&A）任务的应用。主要重点是评估 ChatGPT 在一个精心构建的测试数据集上的表现，并将独立 LLM 的结果与通过检索增强生成（RAG）方法生成的结果进行比较。尽管 LLM 近年来取得了进展，但它们容易生成不正确或“幻觉”信息，这在要求高准确性和可靠性的应用中是一个重大局限。本研究探索了在 LLM 中利用 RAG 的潜力，该方法整合外部知识库和复杂的检索技术，以提高生成输出的准确性和相关性。在此背景下，本文评估了 ChatGPT 回答领域特定问题的能力，采用了两种方法：A）LLM 直接响应，以及 B）LLM 在 RAG 框架内响应。这些方法的有效性通过人工评估和 LLM 评估的双重机制进行评估，对响应的正确性和其他指标进行评分。研究结果强调了在 LLM 中引入 RAG 流水线所带来的性能提升，特别是在为核领域特定查询生成更准确且上下文更恰当的响应方面。此外，本文还重点介绍了在此类专业领域中进一步优化和提高答案质量的替代方法。
英文摘要：
This paper examines the application of ChatGPT, a large language model (LLM), for question-and-answer (Q&A) tasks in the highly specialized field of nuclear data. The primary focus is on evaluating ChatGPT's performance on a curated test dataset, comparing the outcomes of a standalone LLM with those generated through a Retrieval Augmented Generation (RAG) approach. LLMs, despite their recent advancements, are prone to generating incorrect or 'hallucinated' information, which is a significant limitation in applications requiring high accuracy and reliability. This study explores the potential of utilizing RAG in LLMs, a method that integrates external knowledge bases and sophisticated retrieval techniques to enhance the accuracy and relevance of generated outputs. In this context, the paper evaluates ChatGPT's ability to answer domain-specific questions, employing two methodologies: A) direct response from the LLM, and B) response from the LLM within a RAG framework. The effectiveness of these methods is assessed through a dual mechanism of human and LLM evaluation, scoring the responses for correctness and other metrics. The findings underscore the improvement in performance when incorporating a RAG pipeline in an LLM, particularly in generating more accurate and contextually appropriate responses for nuclear domain-specific queries. Additionally, the paper highlights alternative approaches to further refine and improve the quality of answers in such specialized domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
