我们还需要微调分类器吗?将类型化决策模型应用于24篇ACL会议论文
Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers
AI总结:
本文将24篇ACL论文的预测器替换为类型化决策模型Jev,构建TDM-Gauntlet基准,发现零样本提示的Jev可直接替换12个测试集的分类器,提示优化可缩小大部分剩余差距,微调仍最准确。
摘要:
类型化决策模型(Typed Decision Models),例如TypeSafe的Jev(也称为System One或System 1模型),可回答关于输入的类型化问题并给出每个选项的概率,且无需训练。许多NLP论文仍在微调特定任务的分类器,而基于提示的大语言模型(LLM)在大多数文本分类基准上落后于此类分类器。本文研究:在每篇论文自身的评估标准下,一个类型化决策模型可替换多少此类微调分类器,以及在无法替换的场景中需要什么条件。我们将24篇ACL会议论文的预测器替换为Jev,保留每篇论文的划分、标签定义和评估指标,并使用Cloudflare的Clef、Microsoft-Decision-1和OpenAI的GPT-6 Luna Decisions验证结果。由此产生的46个英文测试集(包含其问题和已发表分数)构成了TDM-Gauntlet基准。通过零样本提示,Jev直接在46个测试集中的12个上替换了论文的预测器,这些测试集大多是带有简单标签的单一判断,而在论文方法基于其标签进行训练的场景中则很少成功。在Jev得分较低的场景中,我们将差距归因于标签名称未说明的标注约定。使用500个训练标签加验证,且无需用户管理的GPU,校准和GEPA风格的提示优化可缩小与论文方法之间72%的中位数差距,约为现成GEPA缩小差距的两倍,每次搜索的中位数成本为1.66美元。在相同标签下,微调编码器(如RoBERTa和DeBERTa)可缩小59%的差距,而开源大语言模型Qwen3.8-27B的LoRA微调可缩小97%的差距。微调仍然是最准确的选择,但对于简单标签,零样本提示通常足够,且提示优化可缩小大部分剩余差距。
英文摘要:
Typed Decision Models such as TypeSafe's Jev, also called System One or System 1 models, answer typed questions about an input with a probability for each option, and they need no training. Many NLP papers still fine-tune a task-specific classifier, and prompted large language models (LLMs) trail such classifiers on most text classification benchmarks. We ask how many of these fine-tuned classifiers a Typed Decision Model can replace under each paper's own evaluation, and what is needed where it cannot. We swap the predictors of 24 ACL-venue papers for Jev, keep each paper's splits, label definitions and metric, and check the results with Cloudflare's Clef, Microsoft-Decision-1 and OpenAI's GPT-6 Luna Decisions. The resulting 46 English test sets, with their questions and published scores, form the TDM-Gauntlet benchmark. With a zero-shot prompt, Jev directly replaced the paper's predictor on 12 of the 46 test sets, mostly single judgments with plain labels, but rarely where the paper's method was trained on its labels. Where Jev scored lower, we most often attributed the shortfall to an annotation convention that the label name does not state. With 500 training labels plus validation and no user-managed GPU, calibration and GEPA-style prompt optimization closed a median 72% of the gap to the paper's method, about twice what off-the-shelf GEPA closed, for a median $1.66 per search. On the same labels, fine-tuned encoders such as RoBERTa and DeBERTa closed 59%, and LoRA fine-tuning of the open LLM Qwen3.8-27B closed 97%. Fine-tuning remains the most accurate option, but a zero-shot prompt is often enough for plain labels, and prompt optimization closes most of the remaining gap.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。