MeTMaP:用于检测LLM增强生成中错误向量匹配问题的蜕变测试
MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation
AI总结:
本文提出蜕变测试框架MeTMaP,通过从六个NLP数据集推导八个蜕变关系并生成句子三元组,在203种向量匹配配置上检测LLM增强生成中的错误向量匹配问题,揭示了最高准确率仅41.51%的普遍性缺陷。
摘要:
检索增强生成(RAG)和缓存增强生成(CAG)等增强生成技术通过引入外部知识和缓存信息来增强大语言模型(LLM)输出,已经彻底改变了该领域。然而,作为这些增强技术骨干的向量数据库的集成带来了关键挑战,特别是在确保准确的向量匹配方面。这些数据库中的错误向量匹配会严重损害LLM输出的完整性和可靠性,导致错误信息或错误响应。尽管这些问题影响重大,但在有效检测和解决LLM增强生成中错误向量匹配的方法方面仍存在显著的研究空白。本文提出了MeTMaP,这是一个蜕变测试框架,旨在识别LLM增强生成系统中的错误向量匹配。我们从六个NLP数据集中推导出八个蜕变关系(MR),这些关系构成了我们方法的核心,其基本思想是语义相似的文本应当匹配,而语义不相似的文本不应当匹配。MeTMaP利用这些MR创建句子三元组进行测试,模拟真实的LLM场景。我们在203种向量匹配配置上对MeTMaP进行了评估,涉及29个嵌入模型和7种距离度量,结果揭示了显著的准确性不足。结果显示,与原始数据集相比,我们的测试中最高准确率仅为41.51%,这强调了向量匹配方法中错误匹配问题的普遍性,以及在LLM增强应用中有效检测和缓解的迫切需求。
英文摘要:
Augmented generation techniques such as Retrieval-Augmented Generation (RAG) and Cache-Augmented Generation (CAG) have revolutionized the field by enhancing large language model (LLM) outputs with external knowledge and cached information. However, the integration of vector databases, which serve as a backbone for these augmentations, introduces critical challenges, particularly in ensuring accurate vector matching. False vector matching in these databases can significantly compromise the integrity and reliability of LLM outputs, leading to misinformation or erroneous responses. Despite the crucial impact of these issues, there is a notable research gap in methods to effectively detect and address false vector matches in LLM-augmented generation. This paper presents MeTMaP, a metamorphic testing framework developed to identify false vector matching in LLM-augmented generation systems. We derive eight metamorphic relations (MRs) from six NLP datasets, which form our method's core, based on the idea that semantically similar texts should match and dissimilar ones should not. MeTMaP uses these MRs to create sentence triplets for testing, simulating real-world LLM scenarios. Our evaluation of MeTMaP over 203 vector matching configurations, involving 29 embedding models and 7 distance metrics, uncovers significant inaccuracies. The results, showing a maximum accuracy of only 41.51\% on our tests compared to the original datasets, emphasize the widespread issue of false matches in vector matching methods and the critical need for effective detection and mitigation in LLM-augmented applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。