TRACER:基于追踪的自适应成本高效路由用于LLM分类
TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification
AI总结:
TRACER通过利用生产日志中的标注输入输出对训练轻量级替代模型,实现低成本的LLM分类路由,其通过阈值机制动态决定模型部署边界,并生成可解释性特征。
摘要:
每次调用LLM分类端点都会产生已保存在生产日志中的标注输入输出对。这些对构成一个免费且增长的训练集:一个轻量级替代模型在这些数据上训练后,可以以接近零边际推理成本吸收大量未来流量。开放性问题包括何时替代模型可靠到可以部署,它处理什么情况以及何时延迟,以及随着数据积累,这个边界如何演变。我们介绍了TRACER(基于追踪的自适应成本高效路由)系统,该系统在LLM自身的生产追踪上训练机器学习替代模型,并通过平衡门机制进行部署:只有当替代模型与LLM的一致性超过用户指定的阈值α时,才会激活。为使路由边界透明,TRACER生成可解释性特征,描述替代模型处理哪些输入区域,其何时达到平台期以及为何延迟。在77类意图基准测试中,TRACER根据质量目标α实现83-100%的替代模型覆盖率;在150类基准测试中,替代模型完全取代教师模型。在自然语言推理任务中,平衡门正确拒绝部署,因为嵌入表示无法支持可靠分离。该系统作为开源软件提供。
英文摘要:
Every call to an LLM classification endpoint produces a labeled input-output pair already retained in production logs. These pairs constitute a free, growing training set: a lightweight surrogate trained on them can absorb a significant portion of future traffic at near-zero marginal inference cost. The open questions are when the surrogate is reliable enough to deploy, what it handles versus defers, and how that boundary evolves as data accumulates. We introduce TRACER (Trace-based Adaptive Cost-Efficient Routing), an open-source system that trains ML surrogates on an LLM's own production traces and governs deployment through a parity gate: the surrogate is activated only when its agreement with the LLM exceeds a user-specified threshold α. To make the routing boundary transparent, TRACER generates interpretability artifacts describing which input regions the surrogate handles, where it plateaus, and why it defers. On a 77-class intent benchmark with a Sonnet 4.6 teacher, TRACER achieves 83-100% surrogate coverage depending on the quality target α; on a 150-class benchmark, the surrogate fully replaces the teacher. On a natural language inference task, the parity gate correctly refuses deployment because the embedding representation cannot support reliable separation. The system is available as open-source software.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。