arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.13009cs.CLcs.AI
收藏

MetaReflection:利用过往反思为语言智能体学习指令

MetaReflection: Learning Instructions for Language Agents using Past Reflections

Microsoft(微软公司)

Priyanshu Gupta, Shashank Kirtania, Ananya Singha, Sumit Gulwani, Arjun Radhakrishna, Sherry Shi, Gustavo Soares

AI总结:

针对闭源LLM驱动的语言智能体难优化、现有方法适用场景受限的问题,提出离线强化学习方法MetaReflection,通过扩充经验语义记忆提升性能,在多任务上效果优于GPT-4基线,且调用成本更低。

摘要:

大语言模型(LLMs)的普及开启了语言智能体解决各类任务的新时代。尽管当前前沿的LLMs足以支撑性能尚可的语言智能体,但闭源API模式使得在其表现不佳时难以进行优化。为解决这一问题,近期研究探索了利用自反思、提示优化等技术提升其性能的方法。遗憾的是,自反思等技术仅能在在线场景中使用,而当前的提示优化技术仅针对简单任务设计和测试。为此,我们提出MetaReflection——一种新型离线强化学习技术,它通过基于过往试验的经验学习扩充语义记忆,从而提升语言智能体的性能。我们在多个领域验证了MetaReflection的有效性,涵盖复杂逻辑推理、生物医学语义相似度、开放世界问答,以及基础设施即代码场景下的漏洞威胁检测,涉及不同的智能体设计。相较于原始GPT-4基线,MetaReflection将语言智能体的性能提升了4%至16.82%,性能与现有最先进的提示优化技术相当,且所需的LLM调用次数更少。

英文摘要:

The popularity of Large Language Models (LLMs) have unleashed a new age ofLanguage Agents for solving a diverse range of tasks. While contemporary frontier LLMs are capable enough to power reasonably good Language agents, the closed-API model makes it hard to improve in cases they perform sub-optimally. To address this, recent works have explored ways to improve their performance using techniques like self-reflection and prompt optimization. Unfortunately, techniques like self-reflection can be used only in an online setup, while contemporary prompt optimization techniques are designed and tested to work on simple tasks. To this end, we introduce MetaReflection, a novel offline reinforcement learning technique that enhances the performance of Language Agents by augmenting a semantic memory based on experiential learnings from past trials. We demonstrate the efficacy of MetaReflection by evaluating across multiple domains, including complex logical reasoning, biomedical semantic similarity, open world question answering, and vulnerability threat detection, in Infrastructure-as-Code, spanning different agent designs. MetaReflection boosts Language agents' performance by 4% to 16.82% over the raw GPT-4 baseline and performs on par with existing state-of-the-art prompt optimization techniques while requiring fewer LLM calls.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑