arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.19558cs.CL
收藏

TextReasoningBench: 推理是否真的能提升大语言模型中的文本分类性能?

TextReasoningBench: Does Reasoning Really Improve Text Classification in Large Language Models?

School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院);School of Nursing, The Hong Kong Polytechnic University(香港理工大学护理学院)

Xinyu Guo, Yazhou Zhang, Jing Qin

AI总结:

本文通过TextReasoningBench评估推理策略在文本分类中的有效性与效率,发现推理并非总能提升性能,且多数策略效率低下。

摘要:

从大语言模型(LLMs)中提取显式的、分步的推理轨迹已成为增强模型能力的主要范式。尽管此类推理策略最初设计用于需要显式多步推理的问题,但它们已越来越多地应用于各种NLP任务。这种扩展隐含假设了反思性推理对异构任务具有普遍益处。然而,推理机制是否真正有助于分类任务仍鲜有研究,尤其是在考虑其显著的token和时间成本的情况下。为填补这一空白,我们引入了TextReasoningBench,一个系统性的基准,用于评估推理策略在LLMs文本分类中的有效性和效率。我们比较了七种推理策略(IO、CoT、SC-CoT、ToT、GoT、BoC和long-CoT)在十个LLMs和五个文本分类数据集上的表现。除了传统指标如准确率和宏F1之外,我们还引入了两种成本感知的评估指标,量化每token推理的性能增益以及相对于token成本增长的性能改进效率。实验结果揭示了三个显著发现:(1)推理并不总是提升分类性能:尽管中等策略如CoT和SC-CoT产生一致但有限的收益(通常在大模型上增加+1%至+3%),更复杂的策略(如ToT和GoT)往往无法超越更简单的基线,并且可能在小型模型上退化;(2)推理通常效率低下:许多推理策略使token消耗增加10倍至100倍(例如SC-CoT和ToT),而仅提供微小的性能提升。

英文摘要:

Eliciting explicit, step-by-step reasoning traces from large language models (LLMs) has emerged as a dominant paradigm for enhancing model capabilities. Although such reasoning strategies were originally designed for problems requiring explicit multi-step reasoning, they have increasingly been applied to a broad range of NLP tasks. This expansion implicitly assumes that deliberative reasoning uniformly benefits heterogeneous tasks. However, whether such reasoning mechanisms truly benefit classification tasks remains largely underexplored, especially considering their substantial token and time costs. To fill this gap, we introduce TextReasoningBench, a systematic benchmark designed to evaluate the effectiveness and efficiency of reasoning strategies for text classification with LLMs. We compare seven reasoning strategies, namely IO, CoT, SC-CoT, ToT, GoT, BoC, and long-CoT across ten LLMs on five text classification datasets. Beyond traditional metrics such as accuracy and macro-F1, we introduce two cost-aware evaluation metrics that quantify the performance gain per reasoning token and the efficiency of performance improvement relative to token cost growth. Experimental results reveal three notable findings: (1) Reasoning does not universally improve classification performance: while moderate strategies such as CoT and SC-CoT yield consistent but limited gains (typically +1% to +3% on big models), more complex methods (e.g., ToT and GoT) often fail to outperform simpler baselines and can even degrade performance, especially on small models; (2) Reasoning is often inefficient: many reasoning strategies increase token consumption by 10$\times$ to 100$\times$ (e.g., SC-CoT and ToT) while providing only marginal performance improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑