逐Token重生成与领域偏差:大型语言模型在高级数学问题求解上的基准测试
Token-by-Token Regeneration and Domain Biases: A Benchmark of LLMs on Advanced Mathematical Problem-Solving
AI总结:
本研究评估了10个7至8亿参数的LLMs在945道竞赛级数学题上的表现,发现商业模型与开源模型存在34.5%的性能差距,并指出逐token重生成虽略微提升准确率但显著增加执行时间,混合推理方法或为未来方向。
摘要:
大型语言模型(LLMs)在许多自然语言任务中表现出色,但在复杂的数学问题求解方面仍存在困难,特别是在符号推理和保持输出一致性方面。本研究使用来自MATH数据集的945道竞赛级问题,评估了10个具有7至8亿参数的LLMs。研究重点在于它们在推理过程中生成可执行Python代码的能力,涉及超过9,450次代码执行。该研究引入了一个评估框架,使用mistral-large-2411以5分制对答案进行评分,这有助于解决数学符号中的不一致问题。研究还探讨了逐token重生成输出对优化结果的影响。研究结果显示,表现最好的商业模型(gpt-4o-mini,得分83.7%)与效果最差的开源模型(open-codestral-mamba:v0.1,得分49.2%)之间存在34.5%的显著性能差距。这种差异在数论等复杂领域尤为明显。虽然逐token重生成略微提高了llama3.1:8b模型的准确率(+0.8%),但同时也使代码执行时间减少了36.7%,凸显了效率与精度之间的权衡。研究还观察到,所有模型中普遍存在难度越高准确率越低的趋势。尽管使用了受控执行环境,生成的代码中仍有不到1%是不安全的,并且在10次尝试后仍有3.17%的问题未解决,这表明混合推理方法可能是有益的。
英文摘要:
Large language models (LLMs) excel in many natural language tasks, yet they struggle with complex mathemat-ical problem-solving, particularly in symbolic reasoning and maintaining consistent output. This study evalu-ates 10 LLMs with 7 to 8 billion parameters using 945 competition-level problems from the MATH dataset. The focus is on their ability to generate executable Python code as a step in their reasoning process, involving over 9,450 code executions. The research introduces an evaluation framework using mistral-large-2411 to rate answers on a 5-point scale, which helps address inconsistencies in mathematical notation. It also examines the impact of regenerating output token-by-token on refining results. The findings reveal a significant 34.5% per-formance gap between the top commercial model (gpt-4o-mini, scoring 83.7%) and the least effective open-source model (open-codestral-mamba:v0.1, scoring 49.2%). This disparity is especially noticeable in complex areas like Number Theory. While token-by-token regeneration slightly improved accuracy (+0.8%) for the model llama3.1:8b, it also reduced code execution time by 36.7%, highlighting a trade-off between efficiency and precision. The study also noted a consistent trend where harder problems correlated with lower accuracy across all models. Despite using controlled execution environments, less than 1% of the generated code was unsafe, and 3.17% of problems remained unsolved after 10 attempts, suggesting that hybrid reasoning methods may be beneficial.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。