流行度偏移下的类型化决策模型:决策训练、校准与隐式先验
Typed Decision Models under Prevalence Shift and Label Budgets: Decision Training, Calibration, and Implicit Priors
AI总结:
该研究对比Jev等3种类型化决策模型、其骨干模型及微调分类器在流行度偏移下的表现,发现决策训练并非始终有益,Jev的校准度在偏移下仅在金融情感任务中弱于微调DeBERTa,EM先验校正可降低其偏移下的校准误差。
摘要:
类型化决策模型(Typed Decision Models, TDMs)会返回调用者定义选项的概率,其供应商称Jev是经过校准的。但在部署时类别频率会发生变化,因此在固定测试集上测得的校准度在应用阈值的场景中未必成立。现有评估存在两个缺口:一是在相同读出机制下,供应商的决策训练收益未与骨干模型的收益分离;二是TDMs仅在测试集的类别分布下接受过测试。我们探究决策训练是否能提升模型相对于其自身骨干的性能,以及TDMs的校准度和阈值是否能像微调分类器一样承受流行度(标签)偏移。我们在5个公开数据集的模拟偏移场景下,对3种TDMs(Jev、Tev1-4B、Nimble-9B)、它们的基础模型以及2种训练后的分类器的相同决策进行比较,在计算任何测试前固定了35次配对测试。结果显示:第一,决策训练并非始终有益:仅通过LoRA适配器与Qwen3.5-9B不同的Nimble-9B,在意图任务上校准度更好,但在金融情感任务上的Macro-F1低12.0个百分点;第二,在中等偏移下,仅在金融情感任务中,Jev的预期校准误差(ECE)和阈值风险比微调后的DeBERTa显著增长,分别增长1.7和2.5个百分点;在2000次偏移场景(探索性)上取平均,每个数据集上的差距最多为1个百分点;第三,以模型的平均开发预测作为先验的EM先验校正无需标签,可在ECE最高的两个数据集上将Jev在偏移下的ECE降低2.7和1.8个百分点,尽管带标签的重新校准能将其降低更多。我们测试的这3种TDMs既未始终优于其骨干模型,也未在偏移下格外脆弱;在校准度方面,它们的主要弱点是无偏移时已存在的校准误差。我们发布了所有输入、代码和输出。
英文摘要:
Typed decision models (TDMs) return probabilities over caller-defined options, and Jev's vendor describes it as calibrated. We ask whether vendor decision training improves a model over its backbone, whether a TDM's calibration and thresholds withstand prevalence (label) shift like a fine-tuned classifier's, and how many labels a trained model needs to overtake a TDM. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and trained classifiers on identical candidate-set decisions from five public datasets (on intent datasets, within the gold label's intent group) under simulated shift, with 35 paired tests frozen before the confirmatory analysis (earlier test-data checks disclosed), and estimate label-budget curves. First, decision training is not always beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12 Macro-F1 points worse on financial sentiment, a loss that persists across prompt formats; Tev1-4B's largest gain over its backbone is a prompt-format effect. Second, on 50 fixed moderate shifts, Jev's top-label calibration error and threshold risk grow significantly more than a fine-tuned DeBERTa's only on financial sentiment, sensitive to the prior set; averaged over 2,000 shifts the gap is at most one point. Third, label-free EM correction with the model's implicit prior lowers Jev's shifted calibration error where it is highest, but labelled recalibration lowers it far more. Fourth, DeBERTa needs 100-250 labels to overtake Jev on hate speech and does not clearly overtake it on CLINC150 at any budget; a QLoRA adapter on Qwen3.5-4B overtakes Jev with at most 2,500 labels on every dataset, on CLINC150 only marginally and not under in-budget selection. The TDMs tested are neither uniformly better than their backbones nor uniquely fragile under shift; their lead over trained models depends on the task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。