arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.13050cs.CL
收藏

Think-then-Act:一种双角度评估的检索增强生成方法

Think-then-Act: A Dual-Angle Evaluated Retrieval-Augmented Generation

Xi’an Jiaotong University(西安交通大学)

Yige Shen, Hao Jiang, Hua Qu, Jihong Zhao

AI总结:

针对大语言模型存在的时间错位、幻觉等问题,提出双角度评估的检索增强生成框架Think-then-Act,通过两阶段判断减少不必要的检索与查询重写,在多数据集上实现准确率和效率均优于现有基线,还具备资源优化优势。

摘要:

尽管大语言模型(LLM)能力出众,但它们常面临时间错位、生成幻觉内容等挑战。通过检索机制增强LLM以从外部来源获取相关信息是一种颇具前景的解决方案。受“三思而后行”这一谚语启发,我们提出了一种双角度评估的检索增强生成框架\textit{Think-then-Act}。现有方法要么不加区分地重写查询、无论是否必要都执行检索,要么先生成临时响应再决定是否进行额外检索,这会增加模型生成成本;与这些方法不同,我们的框架采用两阶段流程:(i)评估输入查询的清晰度与完整性,判断是否需要重写;(ii)评估模型回答该查询的能力,决定是否需要额外检索。在五个数据集上的实验结果表明,\textit{Think-then-Act}框架显著提升了性能。与现有基线相比,我们的框架在准确率和效率上均有显著提升,且在英文和非英文语境下都表现良好。消融研究验证了最优模型置信度阈值,凸显了我们方法在资源优化方面的优势。

英文摘要:

Despite their impressive capabilities, large language models (LLMs) often face challenges such as temporal misalignment and generating hallucinatory content. Enhancing LLMs with retrieval mechanisms to fetch relevant information from external sources offers a promising solution. Inspired by the proverb "Think twice before you act," we propose a dual-angle evaluated retrieval-augmented generation framework \textit{Think-then-Act}. Unlike previous approaches that indiscriminately rewrite queries or perform retrieval regardless of necessity, or generate temporary responses before deciding on additional retrieval, which increases model generation costs, our framework employs a two-phase process: (i) assessing the input query for clarity and completeness to determine if rewriting is necessary; and (ii) evaluating the model's capability to answer the query and deciding if additional retrieval is needed. Experimental results on five datasets show that the \textit{Think-then-Act} framework significantly improves performance. Our framework demonstrates notable improvements in accuracy and efficiency compared to existing baselines and performs well in both English and non-English contexts. Ablation studies validate the optimal model confidence threshold, highlighting the resource optimization benefits of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑