arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.06370cs.SEcs.AI
收藏

面向分析和改进启用大语言模型(LLM)的软件的概率框架

Towards a Probabilistic Framework for Analyzing and Improving LLM-Enabled Software

Universidad de Buenos Aires(布宜诺斯艾利斯大学);CONICET(阿根廷国家科学技术研究委员会)

Juan Manuel Baldonado, Flavia Bonomo-Braberman, Víctor Adrián Braberman

AI总结:

针对软件工程中启用LLM的系统可靠性与可验证性挑战,提出概率框架,通过建模语义等效输出簇分布分析改进系统,应用于自动形式化问题,助力识别弱点并引导对齐改进,提升系统可靠性。

摘要:

确保启用大语言模型(LLM)的系统的可靠性和可验证性仍是软件工程中的重大挑战。我们提出一个概率框架,通过建模和细化语义等效输出簇的分布,系统分析和改进这些系统。该框架便于评估和迭代改进迁移模型——利用LLM将输入转换为下游任务输出的关键软件组件。为说明其效用,我们将框架应用于自动形式化问题(将自然语言文档转换为正式程序规范)。案例表明,分布感知分析能识别弱点、引导针对性对齐改进,产出更可靠且可解释的输出。此原则性方法为解决鲁棒LLM启用系统开发中的关键挑战提供基础。

英文摘要:

Ensuring the reliability and verifiability of large language model (LLM)-enabled systems remains a significant challenge in software engineering. We propose a probabilistic framework for systematically analyzing and improving these systems by modeling and refining distributions over clusters of semantically equivalent outputs. This framework facilitates the evaluation and iterative improvement of Transference Models--key software components that utilize LLMs to transform inputs into outputs for downstream tasks. To illustrate its utility, we apply the framework to the autoformalization problem, where natural language documentation is transformed into formal program specifications. Our case illustrates how distribution-aware analysis enables the identification of weaknesses and guides focused alignment improvements, resulting in more reliable and interpretable outputs. This principled approach offers a foundation for addressing critical challenges in the development of robust LLM-enabled systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑