arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.02543cs.LG
收藏

推理时蒸馏:无需微调或手动提示工程的低成本代理

Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineering

Stanford University(斯坦福大学)

Vishnu Sarukkai, Asanshay Gupta, James Hong, Michaël Gharbi, Kayvon Fatahalian

AI总结:

本文提出通过推理时技术实现低成本高准确率的代理,无需微调或手动提示工程,在ALFWorld和AppWorld上分别实现2.5倍和3.5倍的成本降低,同时保持高准确率。

摘要:

在大规模部署LLM代理时,通常需要在质量和成本之间做出选择。现有成本降低方法无法保持敏捷性:快速迭代而无需人力瓶颈的能力。提示工程是脆弱且缓慢迭代的,而微调需要多天训练和固定设计的承诺;两者都不适合迭代工作流和时间敏感的批量任务。我们证明,已建立的推理时技术——动态上下文学习和自我一致性级联——可以用来移动成本-准确性帕累托前沿,同时保持敏捷性。实践者在小任务子集上运行教师模型以收集演示,然后立即部署更便宜的学生模型处理剩余部分。在每一步,系统检索相关的教师演示作为上下文示例。当多个学生样本一致时,我们继续;当它们分歧时,我们回退到教师。这不需要提示工程或训练。在ALFWorld上,我们以2.5倍更低的成本(0.059到0.024每集)匹配教师准确性。在AppWorld上,我们实现了3.5倍的成本降低,同时恢复79%的教师准确性。我们的实证分析提供了关键设计选择的指导:教师数据库大小、演示集大小、检索策略和级联阈值。这些分析突显了推理时的杠杆点,可以在不牺牲人类开发速度的情况下导航成本-性能权衡。

英文摘要:

Deploying LLM agents at scale typically requires choosing between quality and cost. Existing cost-reduction approaches fail to preserve agility: the ability to iterate rapidly without human time bottlenecks. Prompt engineering is brittle and slows iteration, while fine-tuning requires multi-day training and commitment to fixed designs; both are impractical for iterative workflows and time-sensitive batch jobs. We demonstrate that established inference-time techniques--dynamic in-context learning and self-consistency cascades--can be leveraged to shift the cost-accuracy Pareto frontier while preserving agility. Practitioners run the teacher on a small task subset to collect demonstrations, then immediately deploy a cheaper student on the remainder. At each step, the system retrieves relevant teacher demonstrations as in-context examples. When multiple student samples agree, we proceed; when they diverge, we fall back to the teacher. This requires no prompt engineering or training. On ALFWorld, we match teacher accuracy at 2.5x lower cost (0.059 to 0.024 per episode). On AppWorld, we achieve 3.5x cost reduction while recovering 79% of teacher accuracy. Our empirical analyses provide guidance on key design choices: teacher database size, demonstration set size, retrieval strategy, and cascade thresholds. These analyses highlight inference-time levers for navigating cost-performance tradeoffs without sacrificing human development speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑