设计由语言模型赋能的算法:分析框架、案例研究与见解
Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
AI总结:
本文提出了一个用于设计和分析基于LLM的算法的系统分析框架,通过涵盖多种算法模式的案例研究,展示了该框架如何分析任务分解和提示等关键设计选择对算法准确性与效率的影响,并得出了可跨场景推广的见解。
摘要:
本工作提出了一个用于设计和分析基于LLM的算法的分析框架,即包含一次或多次大型语言模型(LLM)调用作为子程序,并严重依赖LLM能力的算法。虽然此类算法,从带有提示工程的基本LLM调用到复杂的LLM驱动的智能体工作流和复合AI系统,已取得了显著的实证成功,但其设计和优化往往需要大量的试错和逐案分析。我们提出的框架旨在缓解这些难题,提供了一种正式且系统的方法,用于分析基于LLM的算法的准确性和效率将如何受到关键设计选择的影响,例如任务分解的模式和粒度,或每次LLM调用的提示。通过涵盖多种算法模式(包括并行/层次/递归任务分解和通用有向无环图)的大量案例研究,我们展示了该框架的实际应用,并得出了可跨场景推广的有趣见解,同时在合成环境中进行了系统的实证验证。
英文摘要:
This work presents an analytical framework for the design and analysis of LLM-based algorithms, i.e., algorithms that contain one or multiple calls of large language models (LLMs) as sub-routines and critically rely on the capabilities of LLMs. While such algorithms, ranging from basic LLM calls with prompt engineering to complicated LLM-powered agentic workflows and compound AI systems, have achieved remarkable empirical success, their design and optimization oftentimes require extensive trial-and-errors and case-by-case analysis. Our proposed framework serves as an attempt to mitigate such headaches, offering a formal and systematic approach for analyzing how the accuracy and efficiency of an LLM-based algorithm will be impacted by critical design choices, such as the pattern and granularity of task decomposition, or the prompt for each LLM call. Through a wide range of case studies covering diverse algorithm patterns (including parallel/hierarchical/recursive task decomposition and generic directed acyclic graphs), we demonstrate the proposed framework in action and derive interesting insights that generalize across scenarios, accompanied by systematic empirical validation in synthetic settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。