元数据推理的代理方法
An Agentic Approach to Metadata Reasoning
AI总结：
本文提出元数据推理代理，通过检索候选表和自主咨询元数据，有效筛选数据源。在KramaBench和合成基准测试中，其在数据选择和避免低质量数据方面表现优异。
摘要：
随着LLM驱动的自主代理执行复杂多步骤任务，发现相关数据源成为关键瓶颈。本文引入元数据推理代理，通过表格搜索引擎检索候选表，并自主咨询元数据以确定候选表是否符合任务要求。通过实证研究，该方法在KramaBench数据集上达到83.16%的平均F1分数，优于现有基线32个百分点。在基于BIRD数据湖的新合成基准测试中，该方法在冗余和低质量表环境中仍能保持85.5%的F1分数，并以99%的成功率避免低质量数据。
英文摘要：
As autonomous LLM agents evolve to perform complex analytical tasks that require integrating multiple datasets, the problem of discovering relevant data sources becomes a key bottleneck. Beyond the challenge posed by the sheer volume of available data sources, data-source selection is difficult because the semantics of data are nuanced and require considering many aspects of the data. To address this challenge, we introduce Metadata Reasoner, an agentic approach to metadata reasoning that identifies a small set of data sources that are both sufficient and minimal for a given analytical task. Metadata Reasoner leverages a table-search engine to retrieve candidate tables, and then autonomously consults various aspects of the available metadata to determine whether the candidates fit the requirements of the task. We demonstrate the effectiveness of Metadata Reasoner through a series of empirical studies. Evaluated on the real-world KramaBench datasets for data selection, our approach achieves an average F1 score of 88.63%, outperforming the strongest baseline by 10.73 percentage points. Furthermore, evaluations on a newly created synthetic benchmark based on the BIRD dataset reveal that Metadata Reasoner is highly robust against redundant and low-quality tables that are in the data lake. In this noisy environment, Metadata Reasoner maintains a 93.5% average F1 score for selecting the right datasets and demonstrates a 100% success rate in avoiding low-quality data. Finally, we extend Metadata Reasoner to support the interleaving of table selection with the execution of partial queries. Compared with a single pass of table selection, such interleaving improves table-selection F1 score on runtime-dependent tasks by 4.8 percentage points and end-to-end task accuracy by 5.3 points.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
