arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.05337cs.CL
收藏

使用LLM解释竞赛级编程解决方案

Explaining Competitive-Level Programming Solutions using LLMs

The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Warsaw(华沙大学);IDEAS NCBR(IDEAS国家研究中心);University of Houston(休斯顿大学)

Jierui Li, Szymon Tworkowski, Yingying Wu, Raymond Mooney

AI总结:

该研究提出自动为竞赛编程的问题-方案对生成结构化自然语言解释的方法,并在CodeContests上验证其专家满意度及辅助LLM解题的效果。

摘要:

本文将竞赛级编程问题求解视为一项结合推理与代码生成的复合任务。我们提出一种新颖方法,可自动为<problem, solution>(问题,解决方案)对标注自然语言解释。研究表明,尽管当前最先进的LLM在求解竞赛级编程问题时表现较差,但它们在描述和解释解决方案方面展现出很强能力。我们的解释生成方法能够为问题生成结构化解决方案解释,其中包含描述与分析。为评估标注解释的质量,我们从两个方面考察其有效性:1)让编写标准答案方案的人类编程专家满意;2)帮助LLM更有效地求解问题。在CodeContests数据集上的实验结果表明,尽管GPT3.5与GPT-4描述解决方案的能力相当,但GPT-4对解决方案背后关键思想的理解更好。

英文摘要:

In this paper, we approach competitive-level programming problem-solving as a composite task of reasoning and code generation. We propose a novel method to automatically annotate natural language explanations to \textit{<problem, solution>} pairs. We show that despite poor performance in solving competitive-level programming problems, state-of-the-art LLMs exhibit a strong capacity in describing and explaining solutions. Our explanation generation methodology can generate a structured solution explanation for the problem containing descriptions and analysis. To evaluate the quality of the annotated explanations, we examine their effectiveness in two aspects: 1) satisfying the human programming expert who authored the oracle solution, and 2) aiding LLMs in solving problems more effectively. The experimental results on the CodeContests dataset demonstrate that while LLM GPT3.5's and GPT-4's abilities in describing the solution are comparable, GPT-4 shows a better understanding of the key idea behind the solution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑