Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
Emory University(埃默里大学);University of Tuebingen(图宾根大学);University of Surrey(萨里大学);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所)
Xinyuan Song, Keyu Wang, PengXiang Li, Lu Yin, Shiwei Liu
Recent studies suggest that the deeper layers of Large Language Models (LLMs) contribute little to representation learning and can often be removed without significant performance loss. However, such claims are typically drawn from narrow evaluations and may overlook important aspects of model behavior. In this work, we present a systematic study of depth utilization across diverse dimensions, including evaluation protocols, task categories, and model architectures. Our analysis confirms that very deep layers are generally less effective than earlier ones, but their contributions vary substantially with the evaluation setting. Under likelihood-based metrics without generation, pruning most layers preserves performance, with only the initial few being critical. By contrast, generation-based evaluation uncovers indispensable roles for middle and deeper layers in enabling reasoning and maintaining long-range coherence. We further find that knowledge and retrieval are concentrated in shallow components, whereas reasoning accuracy relies heavily on deeper layers -- yet can be reshaped through distillation. These results highlight that depth usage in LLMs is highly heterogeneous and context-dependent, underscoring the need for task-, metric-, and model-aware perspectives in both interpreting and compressing large models.
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University College London, University of London(伦敦大学大学学院);Dalian University of Technology(大连理工大学)
HalluGuard: Demystifying Data-Driven and Reasoning-Driven Hallucinations in LLMs
CS Department Virginia Tech(弗吉尼亚理工学院计算机科学系);EECS Department MIT(麻省理工学院电子工程与计算机科学系);CS Department Dartmouth College(达特茅斯学院计算机科学系);Statistics Department Virginia Tech(弗吉尼亚理工学院统计学系);CS Department Michigan State University(密歇根州立大学计算机科学系)
Demystifying Agent Skills: Why They Work-Until They Don't
Princeton University(普林斯顿大学);UC San Diego(加州大学圣迭戈分校);Stanford University(斯坦福大学);University of Southern California(南加利福尼亚大学);Johns Hopkins University(约翰斯·霍普金斯大学)
Improving Context Fidelity via Native Retrieval-Augmented Reasoning
DIRO, Université de Montréal(蒙特利尔大学计算机科学与运筹学系);MetaGPT;Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);Yale University(耶鲁大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能教席)
R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation
The Shenzhen International Graduate School, Tsinghua University, China(清华大学深圳国际研究生院);State Key Laboratory of Nuclear Power Safety Technology and Equipment, China(核能安全技术与装备国家重点实验室);School of Computer Science and Information Engineering, Hefei University of Technology, China(合肥工业大学计算机科学与信息工程学院)
Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization
Zhejiang University(浙江大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);South China University of Technology(华南理工大学);University of British Columbia(不列颠哥伦比亚大学);Fudan University(复旦大学);University of Hong Kong(香港大学)
From Token to Action: State Machine Reasoning to Mitigate Overthinking in Information Retrieval
Computer Science and Engineering, Seoul National University(计算机科学与工程系,首尔国立大学);Interdisciplinary Program in Artificial Intelligence, Seoul National University(人工智能交叉学科项目,首尔国立大学)
ActMem: Bridging the Gap Between Memory Retrieval and Reasoning in LLM Agents
State Key Laboratory for Novel Software Technology, Nanjing University, China(南京大学新型软件技术国家重点实验室);Alibaba Group, Hangzhou, China(阿里巴巴集团,杭州,中国);National Institute of Healthcare Data Science, Nanjing University, China(南京大学健康数据科学国家研究院)
SenseTime Research(商汤科技研究院);Nanyang Technological University(南洋理工大学);University of California, Berkeley(加州大学伯克利分校);University of California, San Diego(加州大学圣地亚哥分校);Carnegie Mellon University(卡内基梅隆大学)
MiRAGE: Misconception Detection with Retrieval-Guided Multi-Stage Reasoning and Ensemble Fusion
Hanoi University of Science and Technology(河内科学技术大学);School of Computing, Phenikaa University(现象学大学计算机学院);Faculty of Interdisciplinary Digital Technology, Phenikaa University(现象学大学跨学科数字技术学院)
MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection
Beijing University of Posts and Telecommunications(北京邮电大学);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);College of Electrical Engineering, Zhejiang University(浙江大学电气工程学院);Zhongguancun Academy(中关村学院);China Telecommunications Corporation(中国电信集团有限公司)