Recent advances in Large Language Models (LLMs) have intensified the debate surrounding the fundamental nature of their reasoning capabilities. While achieving high performance on benchmarks such as GPQA and MMLU, these models exhibit limitations in more complex reasoning tasks, highlighting the need for more rigorous evaluation methodologies. We propose a novel phenomenological approach that goes beyond traditional accuracy metrics to probe the underlying mechanisms of model behavior, establishing a framework that could broadly impact how we analyze and understand AI systems. Using positional bias in multiple-choice reasoning tasks as a case study, we demonstrate how systematic perturbations can reveal fundamental aspects of model decision-making. To analyze these behaviors, we develop two complementary phenomenological models: a Probabilistic Mixture Model (PMM) that decomposes model responses into reasoning, memorization, and guessing components and an Information-Theoretic Consistency (ITC) analysis that quantifies the relationship between model confidence and strategy selection. Through controlled experiments on reasoning benchmarks, we show that true reasoning remains challenging for current models, with apparent success often relying on sophisticated combinations of memorization and pattern matching rather than genuine logical deduction. More fundamentally, we demonstrate that accuracy alone often overstates a model's reasoning abilities, as model behavior can be characterized through underlying mechanisms in the phase space of cognitive strategies, revealing how models dynamically balance different approaches when responding to queries. This framework enables quantitative criteria for real-world deployments, allowing applications to specify reliability thresholds based on strategy distributions rather than aggregate performance metrics.
Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Biochemistry and Molecular Genetics(生物化学与分子遗传学系);University of Illinois Chicago(伊利诺伊大学芝加哥分校)
Quantifying Logical Consistency in Transformers via Query-Key Alignment
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究所);Moscow Institute of Physics and Technology(莫斯科物理技术学院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学);Artificial Intelligence Research Institute (AIRI)(人工智能研究所(AIRI))
RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for Chain-of-Thought Reasoning
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology(社会计算与交互机器人研究中心,哈尔滨工业大学);Faculty of Computing, Harbin Institute of Technology(计算学院,哈尔滨工业大学);School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学);Chinese University of Hong Kong(香港中文大学);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab)
Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Models
LARG, Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology(LARG,社会计算与交互机器人研究中心,哈尔滨工业大学);School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学);The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);ByteDance Seed (China)(字节跳动种子(中国))
Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R$^2$)GRPO
HKUST Hong Kong SAR, China(香港科技大学);SEU Jiangsu, China(江苏大学);HKUST(GZ) Guangzhou, China(香港科技大学(广州));Zhipu AI Beijing, China(智谱AI);HKUST(GZ), HKUST Guangzhou, China(香港科技大学(广州)、香港科技大学)
GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学)
Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models
University of California, Berkeley(加州大学伯克利分校);University of Cambridge(剑桥大学);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Tokyo(东京大学)