本文探讨了前沿大语言模型在体育领域的分析推理能力。我们的分析推理任务具体为：让大语言模型统计NBA（美国职业篮球联赛）和NFL（美国国家橄榄球联盟）比赛中每支球队单节的得分。我们的主要发现分为两方面。首先，在所有测试模型中，GPT-4的表现最为突出，其次是Claude-2.1，GPT-3.5、Gemini-Pro和Llama-2-70b则表现落后。具体而言，我们对比了三种不同的提示技术和一种分治方法，发现分治方法效果最佳。我们的分治方法将逐场比赛的详细数据拆解为更小、更易处理的片段，分别解决每个片段的问题后再汇总结果。除分治方法外，我们还探索了思维链（Chain of Thought, CoT）策略，该策略能显著提升部分模型的表现，尤其是GPT-4和Claude-2.1，二者的准确率均有明显提升；但CoT策略对GPT-3.5、Gemini-Pro等其他模型的性能影响微乎其微，甚至会产生负面效果。其次，令我们意外的是，包括GPT-4在内的大多数模型，尽管在统计NFL单节得分方面表现出色，但在准确统计NBA单节总得分时却存在困难。这促使我们通过大量实验进一步探究影响分析推理任务复杂度的因素，最终得出结论：任务复杂度取决于上下文长度、信息密度以及相关信息的存在情况。本研究为分析推理任务的复杂度研究，以及未来大语言模型的开发方向提供了宝贵见解。
英文摘要：
This paper explores the cutting-edge Large Language Model with analytical reasoning on sports. Our analytical reasoning embodies the tasks of letting large language models count how many points each team scores in a quarter in the NBA and NFL games. Our major discoveries are in two folds. Firstly, we find among all the models we employed, GPT-4 stands out in effectiveness, followed by Claude-2.1, with GPT-3.5, Gemini-Pro, and Llama-2-70b lagging behind. Specifically, we compare three different prompting techniques and a divide-and-conquer approach, we find that the latter was the most effective. Our divide-and-conquer approach breaks down play-by-play data into smaller, more manageable segments, solves each piece individually, and then aggregates them together. Besides the divide-and-conquer approach, we also explore the Chain of Thought (CoT) strategy, which markedly improves outcomes for certain models, notably GPT-4 and Claude-2.1, with their accuracy rates increasing significantly. However, the CoT strategy has negligible or even detrimental effects on the performance of other models like GPT-3.5 and Gemini-Pro. Secondly, to our surprise, we observe that most models, including GPT-4, struggle to accurately count the total scores for NBA quarters despite showing strong performance in counting NFL quarter scores. This leads us to further investigate the factors that impact the complexity of analytical reasoning tasks with extensive experiments, through which we conclude that task complexity depends on the length of context, the information density, and the presence of related information. Our research provides valuable insights into the complexity of analytical reasoning tasks and potential directions for developing future large language models.
Hypothesis Testing Prompting Improves Deductive Reasoning in Large Language Models
Shanghai Jiao Tong University（上海交通大学）；MoE Key Lab of Artificial Intelligence（教育部人工智能重点实验室）；AI Institute（人工智能研究院）；State Key Lab of Advanced Optical Communication System and Network（先进光通信系统与网络国家重点实验室）
InfiR : Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoning
Reallm Labs（Reallm实验室）；The Hong Kong Polytechnic University（香港理工大学）；Zhejiang University（浙江大学）；South China University of Technology（华南理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Dalian University of Technology（大连理工大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；TikTok（字节跳动旗下TikTok）；Amazon（亚马逊）
Unlocking the Working Memory of Large Language Models for Latent Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria（林茨ELLIS单元和LIT AI实验室，机器学习研究所，约翰·凯撒大学林茨，奥地利）；NXAI GmbH, Linz, Austria（NXAI公司，林茨，奥地利）
Med-CoReasoner: Reducing Language Disparities in Medical Reasoning via Language-Informed Co-Reasoning
The University of Tokyo（东京大学）；ETH Zürich（苏黎世联邦理工学院）；National Institute of Informatics（日本信息处理学会）；Siriraj Informatics and Data Innovation Center（Siriraj信息与数据创新中心）；Yale University（耶鲁大学）
When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
Harbin Institute of Technology（哈尔滨工业大学）；Singapore Management University（新加坡管理学院）；Monash University（墨尔本大学）；Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（国立新加坡大学）
Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes
Bosch Center for Artificial Intelligence（博世人工智能中心）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Hochschule der Medien（斯图加特媒体大学）；Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用技术大学）
