Open-Jev 在 CallScreenBench 上的判定：基于小语言模型的校准式单次诈骗筛查
Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model
浏览论文内容
中文总结 AI 辅助
本研究将 JevLite 读出方式应用于诈骗电话筛查，通过 LoRA 微调 Qwen3-4B 实现单次前向传播的校准概率输出，在 CallScreenBench 上达到 AUROC 0.974 且无误报，决策更快、延迟更低，贡献在于应用与综合评估。
中文摘要 AI 辅助
对电话通话进行诈骗筛查需要在每个来电者轮次后的毫秒级时间内给出可信的概率。Jev 风格的类型化判定正好能提供这一点：声明的选项输入，每个选项的校准概率通过单次前向传播输出，且不生成任何文本。我们测试了这种读出方式的开放实现 JevLite，并将其应用于诈骗电话筛查：对 Qwen3-4B 进行 LoRA 微调，使得两个答案标签 logits 上的温度缩放 softmax 输出即为 P(scam)。在 41 个保留的 CallScreenBench 场景（共 577 个每轮决策）中，三种子集成达到了 AUROC 0.974，校准误差为 0.052，在预先登记的 0.02 边际下不劣于 LLM 评判者（MiniMax-M3），对合法电话无任何误报，在相同挂断规则下决策提前 1.14 轮，且单次决策在单张消费级 GPU 上耗时 64.5 毫秒，比微调生成答案的同一骨干模型低 4.9 倍。增益在于读出和校准，而非准确性：微调的 ModernBERT 编码器并未显著更差，配方是在测试集暴露下选择的，且所有来电者均为合成数据。我们不声称架构上的新颖性；贡献在于应用以及报告校准、误报和决策时机（连同 AUROC）的评估。
英文摘要
Screening a phone call for fraud needs a trustworthy probability after every caller turn, in milliseconds. Jev-style typed decisions promise exactly that: declared options go in, one calibrated probability per option comes out of a single forward pass, with no generated text. We test an open implementation of this readout, JevLite, on scam-call screening: Qwen3-4B is LoRA-tuned so that the temperature-scaled softmax over two answer-label logits is P(scam). On 41 held-out CallScreenBench scenarios (577 per-turn decisions) a three-seed ensemble reaches AUROC .974 with calibration error .052, non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin, with no false alarms on legitimate calls, decisions 1.14 turns earlier under the same hang-up rule, and 64.5 ms per decision on one consumer GPU, 4.9x lower than the same backbone fine-tuned to generate its answer. The gain is in the readout and calibration, not accuracy: a fine-tuned ModernBERT encoder is not significantly worse, the recipe was selected with test-set exposure, and all callers are synthetic. We claim no architectural novelty; the contribution is the application and an evaluation reporting calibration, false alarms and decision timing alongside AUROC.
发表机构
- Scam.ai (Reality Inc.)（Scam.ai（Reality Inc.））
机构由 AI 辅助整理，请以论文原文为准。
