arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.13833cs.AIcs.CLcs.ITmath.IT
收藏

论AI模型的推理能力及其量化方法

On the Reasoning Capacity of AI Models and How to Quantify It

XYZ University(XYZ大学);Second institution(第二机构)

Santosh Kumar Radha, Oktay Goktas

AI总结:

本文提出一种现象学框架,通过概率混合模型和信息论一致性分析,揭示LLM在推理任务中依赖记忆与模式匹配而非真正逻辑演绎,并主张以策略分布而非准确率量化推理能力。

摘要:

大型语言模型(LLM)的最新进展加剧了围绕其推理能力根本性质的争论。尽管这些模型在GPQA和MMLU等基准测试中取得了高性能,但它们在更复杂的推理任务中表现出局限性,凸显了对更严格评估方法的需求。我们提出了一种新颖的现象学方法,超越了传统的准确率指标,以探究模型行为的底层机制,建立了一个可能广泛影响我们分析和理解AI系统方式的框架。以多项选择推理任务中的位置偏差为案例研究,我们展示了系统性扰动如何揭示模型决策的基本方面。为了分析这些行为,我们开发了两个互补的现象学模型:一个概率混合模型(PMM),将模型响应分解为推理、记忆和猜测成分;以及一个信息论一致性(ITC)分析,量化模型置信度与策略选择之间的关系。通过在推理基准上的受控实验,我们表明真正的推理对当前模型仍具挑战性,表面上的成功往往依赖于记忆和模式匹配的复杂组合,而非真正的逻辑演绎。更根本的是,我们证明准确率本身常常高估了模型的推理能力,因为模型行为可以通过认知策略相空间中的底层机制来表征,揭示模型在响应查询时如何动态平衡不同方法。该框架为现实世界部署提供了定量标准,允许应用根据策略分布而非总体性能指标来指定可靠性阈值。

英文摘要:

Recent advances in Large Language Models (LLMs) have intensified the debate surrounding the fundamental nature of their reasoning capabilities. While achieving high performance on benchmarks such as GPQA and MMLU, these models exhibit limitations in more complex reasoning tasks, highlighting the need for more rigorous evaluation methodologies. We propose a novel phenomenological approach that goes beyond traditional accuracy metrics to probe the underlying mechanisms of model behavior, establishing a framework that could broadly impact how we analyze and understand AI systems. Using positional bias in multiple-choice reasoning tasks as a case study, we demonstrate how systematic perturbations can reveal fundamental aspects of model decision-making. To analyze these behaviors, we develop two complementary phenomenological models: a Probabilistic Mixture Model (PMM) that decomposes model responses into reasoning, memorization, and guessing components and an Information-Theoretic Consistency (ITC) analysis that quantifies the relationship between model confidence and strategy selection. Through controlled experiments on reasoning benchmarks, we show that true reasoning remains challenging for current models, with apparent success often relying on sophisticated combinations of memorization and pattern matching rather than genuine logical deduction. More fundamentally, we demonstrate that accuracy alone often overstates a model's reasoning abilities, as model behavior can be characterized through underlying mechanisms in the phase space of cognitive strategies, revealing how models dynamically balance different approaches when responding to queries. This framework enables quantitative criteria for real-world deployments, allowing applications to specify reliability thresholds based on strategy distributions rather than aggregate performance metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑