CMDBench：复合AI系统中由粗到细多模态数据发现的基准
CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems
AI总结：
本文提出CMDBench基准，用于评估复合AI系统中多模态数据检索器在真实企业数据平台上的数据发现性能，实验表明检索器设计对任务准确率影响达46%，强调了优化智能体与检索器策略的必要性。
摘要：
采用LLM作为智能体，通过与工具和数据检索器交互来完成知识密集型任务的复合AI系统（CASs），已在数据库和AI社区引起了广泛关注。尽管这些系统有潜力补充企业数据平台中数据分析师的典型分析工作流程，但遗憾的是，CASs同样面临着分析师多年来所遭遇的数据发现挑战——组织内各团队和部门之间形成的多模态数据源孤岛，使得难以识别合适的数据源来完成当前任务。现有的数据发现基准并未对这种数据源的多模态性和多重性进行建模。此外，CASs的基准仅优先评估端到端任务性能。为了推动在真实世界环境中评估CASs内多模态数据检索器数据发现性能的研究，我们提出了CMDBench，这是一个对企业数据平台复杂性进行建模的基准。我们改编了开放领域的现有数据集和基准——从问答和复杂推理任务到结构化数据上的自然语言查询——以评估粗粒度和细粒度的数据发现及任务执行性能。我们的实验揭示了数据检索器设计对下游任务性能的影响——在各种模态、数据源和任务难度下，任务准确率平均下降46%。结果表明，需要开发优化策略来识别合适的LLM智能体和检索器，以在企业数据上高效执行CASs。
英文摘要：
Compound AI systems (CASs) that employ LLMs as agents to accomplish knowledge-intensive tasks via interactions with tools and data retrievers have garnered significant interest within database and AI communities. While these systems have the potential to supplement typical analysis workflows of data analysts in enterprise data platforms, unfortunately, CASs are subject to the same data discovery challenges that analysts have encountered over the years -- silos of multimodal data sources, created across teams and departments within an organization, make it difficult to identify appropriate data sources for accomplishing the task at hand. Existing data discovery benchmarks do not model such multimodality and multiplicity of data sources. Moreover, benchmarks of CASs prioritize only evaluating end-to-end task performance. To catalyze research on evaluating the data discovery performance of multimodal data retrievers in CASs within a real-world setting, we propose CMDBench, a benchmark modeling the complexity of enterprise data platforms. We adapt existing datasets and benchmarks in open-domain -- from question answering and complex reasoning tasks to natural language querying over structured data -- to evaluate coarse- and fine-grained data discovery and task execution performance. Our experiments reveal the impact of data retriever design on downstream task performance -- a 46% drop in task accuracy on average -- across various modalities, data sources, and task difficulty. The results indicate the need to develop optimization strategies to identify appropriate LLM agents and retrievers for efficient execution of CASs over enterprise data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
