arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.29052cs.CLcs.AIcs.IRcs.LG
收藏

我们自己的Jev:一次前向传播中的类型化决策,从内部评判器到开源LoRA模型

Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models

Yuchuan Weng, Chi Liu, Xin Chen, Srinivasan Manoharan

AI总结:

该研究提出基于LoRA的类型化决策方法,构建内部评判器降低智能体编码会话分类成本,在JevBench上35B模型性能接近商业参考,2B模型优于多数开源系统,验证了该模式的有效性与经济性。

摘要:

类型化决策是指将证据、问题和固定选项集映射到每个选项的概率,通过读取语言模型的logits或其隐藏状态上的小型头,在一次前向传播中完成计算,而非生成并解析文本。商业系统Jev推广了这种System 1接口,随后出现了开源复现版本。本文报告了该模式的两种应用:第一部分描述了一个内部System 1评判器,用于对约14.8万个智能体编码会话记录进行分类以构建训练语料库。它替代了前沿LLM评判器,后者每次推理成本约1000美元,且其存储的标签仅在两位评判者达成一致的会话中,与两位评判者的共识匹配率仅为26.9%;而自托管的冻结主干模型搭配秩为16的LoRA适配器和指针头,可在一次前向传播中回答三个类型化问题,在单个GPU上约3分钟内评判1000个会话,按参考费率计算成本约0.06美元,仅为前沿评判器成本的约1%,且所有会话均在组织内部处理。第二部分探究不添加参数的读出机制是否具有竞争力,以及训练一个读出机制是否有帮助。将公开的Eikos方案和约2.5万个决策的语料库应用于另外三个主干模型(1B、2B和35B),我们读取LoRA适配解码器的冻结输出层。在JevBench的111个公开难题上,我们基于这些难题开发的单种子35B模型得分79.3%,与商业参考模型(73.0%,未调整p=0.12)无显著差异,且经Holm–Bonferroni校正后领先于五个开源系统;我们的2B模型领先于排行榜上的所有四个kev检查点。单种子消融实验发现,训练输出层的26个字母行未产生可检测的收益,而在后续语料库版本上重新训练使2B模型的难题级准确率提升了11.7个百分点,这表明小型评估的作用十分有限。

英文摘要:

A typed decision maps evidence, a question and a fixed set of options to a probability for every option, computed in one forward pass, by reading a language model's logits or a small head over its hidden states, instead of generating and parsing text. The commercial system Jev popularised this System~1 interface, and open reproductions have followed. We report two uses of the pattern. \emph{Part I} describes an in-house System~1 judge that triages about 148K agentic coding session records for a training corpus. It replaces a frontier LLM judge that cost on the order of \$1,000 per pass and whose stored labels matched a fresh two-judge consensus on only 26.9\% of the sessions where the two judges agreed; a self-hosted frozen backbone with a rank-16 LoRA adapter and a pointer head answers three typed questions in one forward pass, judges 1,000 sessions in about three minutes of one GPU, about \$0.06 at a reference rate and roughly 1\% of the frontier judge's cost, and keeps every session inside the organisation. \emph{Part II} asks whether a readout that adds no parameters can be competitive and whether training one helps. Applying the public Eikos recipe and corpus of about 25K decisions to three further backbones (1B, 2B and 35B), we read the frozen output layer of a LoRA-adapted decoder. On the 111 public hard items of JevBench our single-seed 35B, developed with those items in view, scores 79.3\%, not significantly different from the commercial reference (73.0\%, unadjusted $p=0.12$) and ahead of five open systems after Holm--Bonferroni correction; our 2B is ahead of all four kev checkpoints on the leaderboard. A single-seed ablation finds no detectable benefit from training the 26 letter rows of the output layer, and a retrain on a later corpus release moves the 2B's hard-tier accuracy by 11.7 points, which shows how little small evaluations resolve.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑