ChatGPT在技术报告理解过程中的不准确性缓解:我们做到了吗?
ChatGPT Inaccuracy Mitigation during Technical Report Understanding: Are We There Yet?
AI总结:
针对ChatGPT在技术报告理解中的幻觉问题,提出CHIME引擎,通过上下文无关文法解析堆栈跟踪并应用蜕变测试和查询转换验证修复回答,比基线提升30.3%修正率。
摘要:
幻觉,即产生不相关/不正确回答的倾向,是基于生成式AI的工具(如ChatGPT)中普遍存在的问题。尽管已对ChatGPT在文本回答中的幻觉进行了研究,但尚不清楚ChatGPT对包含文本和技术术语的技术文本如何产生幻觉。我们调查了47名软件工程师,并从两个OSS项目的错误报告中生成了包含412个问答对的基准。我们发现,基于RAG的ChatGPT(即使用基准问题报告调优的ChatGPT)在生成问题答案时正确率为36.4%,原因有两个:1)在理解代码片段(如堆栈跟踪)中的复杂技术内容方面存在局限性,2)在整合技术术语和文本中表示的上下文方面存在局限性。我们提出了CHIME(ChatGPT不准确性缓解引擎),其基本原理是,如果我们能更好地预处理技术报告并指导ChatGPT中的查询验证过程,我们就能解决观察到的局限性。CHIME使用上下文无关文法(CFG)解析技术报告中的堆栈跟踪。然后,CHIME通过应用蜕变测试和查询转换来验证和修复ChatGPT的回答。在我们的基准中,CHIME比ChatGPT回答的修正率高出30.3%。在一项用户研究中,我们发现使用CHIME改进的回答被认为比不使用CHIME的ChatGPT生成的回答更有用。
英文摘要:
Hallucinations, the tendency to produce irrelevant/incorrect responses, are prevalent concerns in generative AI-based tools like ChatGPT. Although hallucinations in ChatGPT are studied for textual responses, it is unknown how ChatGPT hallucinates for technical texts that contain both textual and technical terms. We surveyed 47 software engineers and produced a benchmark of 412 Q&A pairs from the bug reports of two OSS projects. We find that a RAG-based ChatGPT (i.e., ChatGPT tuned with the benchmark issue reports) is 36.4% correct when producing answers to the questions, due to two reasons 1) limitations to understand complex technical contents in code snippets like stack traces, and 2) limitations to integrate contexts denoted in the technical terms and texts. We present CHIME (ChatGPT Inaccuracy Mitigation Engine) whose underlying principle is that if we can preprocess the technical reports better and guide the query validation process in ChatGPT, we can address the observed limitations. CHIME uses context-free grammar (CFG) to parse stack traces in technical reports. CHIME then verifies and fixes ChatGPT responses by applying metamorphic testing and query transformation. In our benchmark, CHIME shows 30.3% more correction over ChatGPT responses. In a user study, we find that the improved responses with CHIME are considered more useful than those generated from ChatGPT without CHIME.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。