arXiv 2602.149552026-02-16cs.CLcs.SE

接触中心AI中的工具感知规划：通过 lineage 引导的查询分解评估LLMs

Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition

Varun Nathan, Shreyas Guha, Ayush Kumar

AI总结：

本文提出接触中心AI中的工具感知规划框架，通过lineage引导查询分解评估LLMs，揭示工具理解的不足及简计划的优势。

摘要：

我们提出一个领域导向的框架和基准，用于接触中心中的工具感知计划生成，其中回答业务洞察的查询，我们的目标用例，需要将其分解为在结构化工具（Text2SQL（T2S）/Snowflake）和非结构化工具（RAG/transcripts）上可执行的步骤，并带有显式的 depends_on 以实现并行。我们的贡献有三个方面：（i）一个基于参考的计划评估框架，运行在两种模式中——一个跨七个维度的度量评估器（例如，工具提示对齐、查询符合性）和一个一次性评估器；（ii）一个数据整理方法，通过评估器→优化器循环迭代改进计划，以生成高质量的计划lineage（有序的计划修订）同时减少手动工作；（iii）对14个LLM进行大规模研究，涵盖大小和家族，评估它们将查询分解为逐步、可执行和工具分配计划的能力，在带有和不带有lineage的提示下。经验上，LLMs在复合查询和超过4步的计划（通常为5-15步）上表现不佳；最佳总度量得分达到84.8%（Claude-3-7-Sonnet），而最强的一次性匹配率在

英文摘要：

We present a domain-grounded framework and benchmark for tool-aware plan generation in contact centers, where answering a query for business insights, our target use case, requires decomposing it into executable steps over structured tools (Text2SQL (T2S)/Snowflake) and unstructured tools (RAG/transcripts) with explicit depends_on for parallelism. Our contributions are threefold: (i) a reference-based plan evaluation framework operating in two modes - a metric-wise evaluator spanning seven dimensions (e.g., tool-prompt alignment, query adherence) and a one-shot evaluator; (ii) a data curation methodology that iteratively refines plans via an evaluator->optimizer loop to produce high-quality plan lineages (ordered plan revisions) while reducing manual effort; and (iii) a large-scale study of 14 LLMs across sizes and families for their ability to decompose queries into step-by-step, executable, and tool-assigned plans, evaluated under prompts with and without lineage. Empirically, LLMs struggle on compound queries and on plans exceeding 4 steps (typically 5-15); the best total metric score reaches 84.8% (Claude-3-7-Sonnet), while the strongest one-shot match rate at the "A+" tier (Extremely Good, Very Good) is only 49.75% (o3-mini). Plan lineage yields mixed gains overall but benefits several top models and improves step executability for many. Our results highlight persistent gaps in tool-understanding, especially in tool-prompt alignment and tool-usage completeness, and show that shorter, simpler plans are markedly easier. The framework and findings provide a reproducible path for assessing and improving agentic planning with tools for answering data-analysis queries in contact-center settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文