LLM-ProS: 分析大语言模型在竞争性问题解决中的性能
LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving
AI总结:
LLM-ProS评估了多种大语言模型在ICPC问题中的性能,揭示了模型在泛化、适应和解决新问题方面的差异,并探讨了训练方法和数据集对性能的影响。
摘要:
大语言模型的快速发展为自动化复杂问题解决任务,如算法编程和竞赛编程,开辟了新途径。本文介绍了一种新的评估技术,LLM-ProS,用于评估最先进的大语言模型在国际大学生编程竞赛(ICPC)问题上的性能。使用了一个经过精心挑选的包含2011年至2024年世界总决赛问题的166个问题的数据集,我们对模型的推理、准确性和效率进行了基准测试。我们评估了五种模型——GPT-4o、Mistral Large、Llama-3.1-405B以及o1家族(包括o1-mini和o1-preview)在正确性、资源利用和响应校准等关键指标上的表现。我们的结果揭示了模型在泛化、适应和解决新问题方面存在显著差异。我们还研究了训练方法、数据集污染和思维链推理对模型性能的影响。发现为优化大语言模型用于算法任务提供了新的见解,突显了当前模型的优势和局限性。
英文摘要:
The rapid advancement of large language models has opened new avenues for automating complex problem-solving tasks such as algorithmic coding and competitive programming. This paper introduces a novel evaluation technique, LLM-ProS, to assess the performance of state-of-the-art LLMs on International Collegiate Programming Contest (ICPC) problems. Using a curated dataset of 166 World Finals problems from 2011 to 2024, we benchmark the models' reasoning, accuracy, and efficiency. We evaluate the five models-GPT-4o, Mistral Large, Llama-3.1-405B, and the o1 family, consisting of o1-mini and o1-preview, across critical metrics like correctness, resource utilization, and response calibration. Our results reveal significant differences in the models' abilities to generalize, adapt, and solve novel problems. We also investigated the impact of training methodologies, dataset contamination, and chain-of-thought reasoning on model performance. The findings provide new insights into optimizing LLMs for algorithmic tasks, highlighting both strengths and limitations of current models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。