关闭自动研究循环:面向生产搜索排名的AI合作科学家
Closing the Auto-Research Loop: An AI Co-Scientist for Production Search Ranking
AI总结:
提出AI合作科学家框架,通过LLM代理与云计算集成,自动迭代生成想法、实现代码、进行GPU实验并分析结果,在搜索排名任务中带来额外+0.083%离线增益。
摘要:
我们提出了一个AI合作科学家框架,该框架为大型在线旅游平台的生产搜索排名系统关闭了研究循环——将LLM代理与直接云计算访问配对,使得想法生成、代码实现、GPU实验和结果分析能够与人类科学家一起端到端迭代。该框架采用混合代理架构:单一LLM代理处理常规工作,而多LLM共识(GPT-5.2、Gemini Pro 3、Claude Opus 4.5)用于更高风险的决策。在生产排名任务上,人工设计的Transformer基线(V2)相比预Transformer基线(V1)提升了+0.118%;AI合作科学家在V2之上的自动循环贡献了额外的+0.083%,合计离线增益为+0.201%,大约在一周多的挂钟时间内完成(单次运行数值;统计限制在论文中讨论)。最有用的AI提案——统一长序列布局、槽位类型嵌入和多阶段学习率调度——是NLP和视觉领域的标准实践,但之前未出现在我们的生产栈中,这表明LLM代理可以作为排名团队的跨学科连接器。我们还报告了部署背景、负面结果和经验教训。
英文摘要:
We present an AI Co-Scientist framework that closes the research loop for the production search-ranking system of a large online travel platform -- pairing LLM agents with direct cloud-compute access so that idea generation, code implementation, GPU experimentation, and result analysis iterate end-to-end with a human scientist in the loop. The framework uses a hybrid agent architecture: single-LLM agents handle routine work, while multi-LLM consensus (GPT-5.2, Gemini Pro 3, Claude Opus 4.5) is invoked for higher-stakes decisions. On the production ranking task, a human-designed transformer baseline (V2) yielded $+0.118\%$ over a pre-transformer baseline (V1); the AI Co-Scientist's automated loop on top of V2 contributed an additional $+0.083\%$, for a combined $+0.201\%$ offline gain delivered in roughly one extra week of wall-clock time (single-run numbers; statistical limits discussed in the paper). The most useful AI proposals -- unified long-sequence layouts, slot-type embeddings, and multi-phase learning-rate schedules -- are standard practice in NLP and Vision but were absent from our production stack, suggesting that LLM agents can serve as cross-disciplinary connectors for ranking teams. We also report deployment context, negative results, and lessons learned.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。