大型语言模型(LLMs)能做减法吗?
Can LLMs subtract numbers?
AI总结:
本文系统研究了LLMs的减法运算能力,发现其减法准确率远低于加法,主要错误集中在结果为负数时遗漏负号,而指令微调能显著解决此问题。
摘要:
我们对大型语言模型(LLMs)中的减法运算进行了系统性研究。尽管先前的基准测试强调加法和乘法,但减法作为一种非交换运算在结构上具有独特性,相比之下受到的关注却较少。我们在加法和减法问题上评估了来自四个系列的八个预训练LLMs。我们的实验表明,减法的准确率远远落后于加法。我们发现,($a-b$) 的错误集中在 ($a<b$) 的情况下。在这些情况下,LLMs经常能得出正确的数值大小,却遗漏了负号。探测分析表明,LLMs在内部编码了结果是否应为负数的信息,然而这一信息往往未在生成的输出中体现。我们进一步测试了少样本学习和指令微调等知名技术,以观察它们能否提升LLMs的表现。结果表明,尽管少样本提示带来了适度的提升,但指令微调模型在生成负号方面达到了近乎完美的准确率。综合来看,这些发现为LLMs在减法运算中的算术能力局限性及其可恢复性提供了更清晰的刻画。
英文摘要:
We present a systematic study of subtraction in large language models (LLMs). While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively little attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for ($a-b$) are concentrated in cases where ($a<b$). In such cases, LLMs frequently produce the correct magnitude but omit the negative sign. Probing analyses show that LLMs internally encode whether results should be negative, yet this information is often not reflected in generated outputs. We further test well-known techniques such as few-shot learning and instruction-tuning to see if they can improve the LLMs' performance. Our results suggest that while few-shot prompting yields modest gains, the instruction-tuned models achieve near-perfect accuracies in generating the negative sign. Together, these findings provide a clearer characterization of the limitations and recoverability of LLMs' arithmetic capabilities in subtraction.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。