技能不是文档:面向LLM智能体技能路由的查询条件基准与两阶段检索器
Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing
AI总结:
针对LLM智能体技能路由中技能兼容性被忽视的问题,提出Reject-as-Resource Retriever (R3)框架,构建双语基准R3-Skill,并设计两阶段检索系统(R3-Embedding + R3-Reranker)显式建模技能兼容性,显著提升检索效果。
摘要:
LLM智能体通过组合多种技能完成复杂任务,技能检索是智能体的前端阶段。技能检索在监督层面上与传统文档检索有根本不同:Top-K联合正确性不仅取决于每个查询-技能对的语义相关性,还取决于检索到的技能能否协同完成给定查询下的任务。这种“技能兼容性”无法仅从独立相关性推导得出。然而,现有的基于LLM的数据合成流程可以为“哪些技能不应在此查询下联合检索”产生直接监督信号——即LLM自身的拒绝决策——而这一信号通常被视为低质量数据被丢弃。为解决这一差距,我们提出Reject-as-Resource Retriever (R3)并构建R3-Skill,一个面向现实智能体技能路由的双语(中英文)技能检索基准。R3-Skill涵盖四种语言方向,具有接近真实用户请求的查询措辞,并通过多专家交叉检查验证。在R3-Skill上,我们构建了一个两阶段检索系统(R3-Embedding + R3-Reranker),将技能兼容性作为显式训练信号。梯度分析表明,“推开”信号在双编码器中因双边平衡而被稀释,但在交叉编码器中充当无损分级排序监督——这促使将其置于交叉编码器阶段,两个数据集上的消融实验证实了这一点。R3-Embedding + R3-Reranker流水线在R3-Skill上达到Hit@1=0.7714,NDCG@10=0.8327,Set-Compat=0.3525。数据集、训练代码和模型权重已开源,用于智能体技能路由。
英文摘要:
Large language model agents increasingly rely on reusable skills, making skill retrieval a critical front-end component of agent systems. Skill retrieval, however, is not ordinary document retrieval: a useful top-$K$ result must contain individually relevant skills that also form an executable set for the current query. Existing benchmarks and training pipelines largely supervise pairwise relevance and discard the rejection decisions produced when a language model judges a sampled skill combination to be implausible. We introduce R3-Skill, a Chinese--English benchmark that retains these rejections as query-conditioned compatibility supervision. R3-Skill contains 10,246 deduplicated skills, 41,592 accepted queries, and 32,828 rejected annotations across four language directions; all multi-skill test labels were independently reviewed by multiple experts, and 15,962 parseable rejections are organized into an eight-class taxonomy. We further propose a two-stage system composed of R3-Embedding, a multi-positive bi-encoder for large-pool recall, and R3-Reranker, a cross-encoder trained with graded ListNet supervision. Our analysis shows that this signal is stage-dependent, helping cross-encoder reranking while providing no benefit for the tested bi-encoder objective. On R3-Skill, the complete pipeline achieves $75.39\%$ Hit@1, $81.97\%$ NDCG@10, and $33.27\%$ Set-Compat, a $36.6\%$ relative gain over the strongest reranking baseline. It also obtains $83.87\%$ NDCG@10 on SkillRet, demonstrating transfer beyond R3-Skill.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。