Current large language models can perform reasonably well on complex tasks that require step-by-step reasoning with few-shot learning. Are these models applying reasoning skills they have learnt during pre-training and reason outside of their training context, or are they simply memorizing their training corpus at finer granularity and have learnt to better understand their context? To tease apart these possibilities, we introduce ALERT, a benchmark and suite of analyses for assessing language models' reasoning ability comparing pre-trained and finetuned models on complex tasks that require reasoning skills to solve. ALERT provides a test bed to asses any language model on fine-grained reasoning skills, which spans over 20 datasets and covers 10 different reasoning skills. We leverage ALERT to further investigate the role of finetuning. With extensive empirical analysis we find that language models learn more reasoning skills such as textual entailment, abductive reasoning, and analogical reasoning during finetuning stage compared to pretraining state. We also find that when language models are finetuned they tend to overfit to the prompt template, which hurts the robustness of models causing generalization problems.
KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation
Beijing University of Posts and Telecommunications（北京邮电大学）；Tongji University（同济大学）；Multimodal Art Projection Research Community（多模态艺术投影研究社区）；Duke University（杜克大学）；ZTE Corporation（中兴通讯股份有限公司）
Improving Multilingual Math Reasoning for African Languages
The African Research Collective（非洲研究集体）；Masakhane NLP；University of Waterloo（滑铁卢大学）；Mila, McGill University（Mila，麦吉尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能 chair）
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Microsoft Research（微软研究院）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；CUHKSZ（香港中文大学（深圳））；THU（清华大学）
Explicit vs. Implicit Biographies: Evaluating and Adapting LLM Information Extraction on Wikidata-Derived Texts
DISI Departement of Computer Science and Engineering, University of Bologna（博洛尼亚大学计算机科学与工程系）；Automobili Lamborghini SpA（法拉利公司）；Digital Humanities Advanced Research Center（数字人文高级研究中心）；KNAW Humanities Cluster, DHLab（KNAW 人文集群，DHLab）；ETH Zurich, Collegium Helveticum（苏黎世联邦理工学院，瑞士学院）
Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources
DAMO Academy, Alibaba Group（达摩院（阿里巴巴集团））；Nanyang Technological University（南洋理工大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Salesforce Research（Salesforce研究院）；Hupan Lab（湖畔实验室）
SoT: Structured-of-Thought Prompting Guides Multilingual Reasoning in Large Language Models
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education（大数据与人工智能交通 key laboratory（北京交通大学））；School of Computer Science and Technology, Beijing Jiaotong University（计算机科学与技术学院（北京交通大学））；University of Montreal（蒙特利尔大学）
Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Acting
School of Computer Science, Guangdong University of Technology（广东技术大学计算机科学学院）；College of Mathematics and Computer Science, Shantou University（汕头大学数学与计算机科学学院）；Huawei Noah’s Ark Lab, Huawei（华为诺亚实验室）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）
AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent Hunyuan（腾讯混元）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；Amphion Technology Co., Ltd.（Amphion科技有限公司）
