Large Language Models (LLMs) are increasingly deployed as reasoning systems, where reasoning paradigms - such as Chain-of-Thought (CoT) and multi-agent systems (MAS) - play a critical role, yet their relative effectiveness and cost-accuracy trade-offs remain poorly understood. In this work, we conduct a comprehensive and unified evaluation of reasoning paradigms, spanning direct single-model generation, CoT-augmented single-model reasoning, and representative MAS workflows, characterizing their reasoning performance across a diverse suite of closed-form benchmarks. Beyond overall performance, we probe role-specific capability demands in MAS using targeted role isolation analyses, and analyze cost-accuracy trade-offs to identify which MAS workflows offer a favorable balance between cost and accuracy, and which incur prohibitive overhead for marginal gains. We further introduce MIMeBench, a new open-ended benchmark that targets two foundational yet underexplored semantic capabilities - semantic abstraction and contrastive discrimination - thereby providing an alternative evaluation axis beyond closed-form accuracy and enabling fine-grained assessment of semantic competence that is difficult to capture with existing benchmarks. Our results show that increased structural complexity does not consistently lead to improved reasoning performance, with its benefits being highly dependent on the properties and suitability of the reasoning paradigm itself. The codes are released at https://gitcode.com/HIT1920/OpenLLMBench.
A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education(教育部多媒体可信感知与高效计算重点实验室);Institute of Artificial Intelligence(人工智能研究院)
TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Illinois at Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Maryland, College Park(马里兰大学学院公园分校);University of California, San Diego(加州大学圣地亚哥分校)
Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications
From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review
Department of Computer and Network Engineering(计算机与网络工程系);United Arab Emirates University(阿联酋大学);Technology Innovation Institute(技术创新研究院);Eötvös Loránd University(埃斯特哈齐·洛朗大学);Research Institute for Digital Future(数字未来研究院);Khalifa University(卡塔尔大学)
Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulation
Tongji University(同济大学);Hong Kong Polytechnic University(香港理工大学);Shanghai University of Electric and Power(上海电力大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy(广东省人工智能与数字经济实验室);University of Electronic Science and Technology of China(电子科技大学)