arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.37732cs.AIcs.CLcs.LG
收藏

ThinkJev:每一种预算下的单次思考,用于校准的类型化决策

ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions

Fujian Bosssoft Co., Ltd.(福建博思软件股份有限公司)

Xiaolu Ding, Kai Yang, Dongdong Liu

AI总结:

ThinkJev 从混合思考模型构建快慢系统,仅在需思考处启用思考,在 5% 目标错误率下,Qwen3.5-27B 部分决策直接回答、部分思考,准确率接近全思考,且在 JevBench 上表现与现有思考 API 相当。

摘要:

语言模型的许多应用最终会产生类型化决策:从一组封闭选项中进行选择,其概率可供后续操作使用。通过一次前向传播读取这些概率速度很快,但在算术、计数和日期比较等需要多步骤的决策上往往出错。让模型先思考可以修正大部分错误,但每个决策需要花费数秒而非毫秒。为了仅在需要思考的地方付出思考成本,ThinkJev 从一个开放的混合思考模型中构建了快速系统(System 1)和慢速系统(System 2),该模型可直接回答或经过思考后回答。System 1 在直接回答时读取模型对选项字母的概率;System 2 在模型对同一缓存提示进行思考后读取相同字母的概率,因此升级不会重复工作,且两个系统均提供相同选项上的概率。由于可在任意数量思考 token 后读取字母,一次采样思考即可给出所有思考预算下的答案。无需额外训练,ThinkJev 仅在对标记的开发数据进行统计测试保证其接受的答案保持在目标错误率以下时,才让 System 1 单独回答。在 5% 的目标下,Qwen3.5-27B 以这种方式回答了 8359 个测试决策中的 24.5%,实现的错误率为 3.9%(Qwen3.5-4B 和 9B 分别为 6.5% 和 0.0%)。对于其余决策,一旦思考结果足够自信就停止每次思考,达到 81.6% 的准确率,而始终思考的准确率为 82.7%,仅使用 72% 的思考 token。仅从 System 1 的置信度选择何时思考节省的成本较少,因为 System 1 最不确定的决策往往不是思考能修正的决策。在类型化决策服务的独立基准 JevBench 的公开项目上,Qwen3.5-27B 在 29.4% 的决策上进行思考,且准确率与列出的两个思考 API 无差异(探索性研究)。代码和数据生成器可在此 https URL 获取。

英文摘要:

Many uses of language models end in a typed decision: a choice among a closed set of options, whose probabilities code can act on. Reading those probabilities from one forward pass is fast but often wrong on decisions that take several steps, such as arithmetic, counting, and date comparison. Letting the model think first fixes most of these errors, but costs seconds per decision instead of milliseconds. To pay for thinking only where it helps, ThinkJev builds a fast and a slow system from one open hybrid-thinking model that can answer directly or after a thought. System 1 reads the model's probabilities for the option letters when it answers directly; System 2 reads the same letters after the model has thought on the same cached prompt, so escalating repeats no work and both systems give probabilities over the same options. Because the letters can be read after any number of thought tokens, one sampled thought also gives the answer under every thinking budget. Without training, ThinkJev lets System 1 answer alone only where a statistical test on labeled dev data guarantees that its accepted answers stay under a target error rate. At a 5% target, Qwen3.5-27B answers 24.5% of 8359 test decisions this way with a realized error of 3.9% (Qwen3.5-4B and 9B: 6.5% and 0.0%). For the rest, stopping each thought as soon as its answer is confident reaches 81.6% accuracy, against 82.7% for always thinking, with 72% of the thought tokens. Choosing when to think from System 1's confidence alone saves less, because the decisions System 1 is least sure of are often not the ones thinking fixes. On the public items of JevBench, an independent benchmark of typed-decision services, Qwen3.5-27B thinks on 29.4% and is not distinguishable in accuracy from the two thinking APIs listed (exploratory). Code and data generators are available at https://github.com/ding-xiaolu/ThinkJev.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑