发表机构
The Hong Kong University of Science and Technology; The Hong Kong Polytechnic University; University of California, Berkeley（香港科技大学; 香港理工大学; 加州大学伯克利分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究基于Transformer的语言模型在算术任务中的表现，通过分解任务、分析损失收敛等，应用人类策略和方法提升其性能，并经多种验证展示有效性，探索其与人类学习者相似性以增强关键应用中的信任。
AI 中文摘要
基于Transformer的大型语言模型（LLMs）在各种自然语言处理任务中持续取得领先性能。然而，它们在基本算术等看似简单的问题上表现不佳，引发了对模型可靠性、安全性和道德部署的担忧。本研究表明，使用对人类学习者有效的方法可以提高在整数算术任务上训练的普通Transformer模型的性能。首先将算术任务分解为明确的子任务，并对每个子任务进行损失收敛阶分析和消融研究。发现LLMs呈现出与人类学习者相似的学习模式，简单子任务学习速度更快。此外，应用解决问题策略和认知强化方法提高了LLMs的准确性，这表明基于Transformer的LLMs在算术中可能与人类学习者共享认知过程。最后通过显著的准确性改进实验、可视化验证和基于解释的分析全面展示了方法的有效性，探索了基于Transformer的LLMs与人类学习者之间潜在的相似性，增强了对LLMs在关键和高风险应用中的信任。
英文摘要
Transformer-based large language models (LLMs) continue to achieve state-of-the-art performance across various natural language processing tasks. However, their subpar performance on seemingly elementary problems, such as basic arithmetic, raises concerns about model reliability, safety, and ethical deployment. In this study, we demonstrate that the performance of a vanilla Transformer model trained on integer arithmetic tasks can be improved using methods effective for human learners. We begin by decomposing the arithmetic task into well-defined subtasks and conducting loss convergence order analysis together with ablation studies for each subtask. Our findings reveal that LLMs exhibit learning patterns similar to those of human learners, with a faster learning speed for simpler subtasks compared to more complex ones. In addition, we successfully improved the accuracy of LLMs by applying problem-solving strategies and cognitive empowerment methods shown to enhance the performance of human learners. This suggests that transformer-based LLMs may share cognitive processes with human learners in arithmetic. Lastly, we provide a comprehensive demonstration of our method's effectiveness, including significant accuracy improvement experiments, visualization verification, and explanation-based analysis to illuminate the intricacies of LLMs in arithmetic learning. In general, this work explores the potential similarities between transformer-based LLMs and human learners, supported by explainable AI (XAI) verifications, ultimately fostering trust in LLMs for critical and high-stakes applications.
