AgentFAIR：用于地理空间数据集公平性评估的多智能体协作框架

中文摘要 AI 辅助

地理空间数据集支持从城市规划到气候建模等各种应用，但对其公平性合规性进行一致评估具有难度。现有评估器使用不同标准和证据来源，在JavaScript渲染页面或特定存储库标识符方面可能失败。对于来自10个存储库的50个数据集，可用工具的标准化分数标准差平均为15.0个百分点，一个数据集达到30.3。本文提出AgentFAIR，一个将结构化元数据提取与13个特定子原则的语言模型评估器相结合的多智能体框架。每个评估器产生0至3的成熟度分数、引用证据和建议，一个评论家检查证据和一致性并可要求有针对性的重新评估。还给出了各项分数、与四个基线工具的排名相关性、在重复运行子集上子原则的一致性、专家研究结果以及API成本等。这些结果支持可审计性和可行性，但有限的基准、不完整的消融和单模型家族验证限制了关于准确性和泛化性的声明。

英文摘要

Geospatial datasets support applications from urban planning to climate modeling, yet consistent assessment of FAIR compliance is difficult. Existing evaluators use different rubrics and evidence sources and may fail on JavaScript-rendered pages or repository-specific identifiers. For 50 datasets from 10 repositories, the standard deviation of normalized scores across available tools averages 15.0 percentage points and reaches 30.3 for one dataset. Because these outputs are not equivalent measurements, we use them to characterize disagreement and failure modes, not comparative accuracy. We present AgentFAIR, a multi-agent framework combining structured metadata extraction with 13 sub-principle-specific LLM evaluators. Each produces a 0-3 maturity score, cited evidence, and recommendations; a critic checks evidence and consistency and can request targeted re-evaluation. Mean Findability, Accessibility, Interoperability, and Reusability scores are 79.7%, 70.4%, 45.3%, and 72.0%. Rank correlations with four baseline tools range from 0.31 to 0.61; the FAIR-enough comparison is not statistically significant. On a 10-dataset repeated-run subset, sub-principle agreement averages 89% (standard deviation: 3 percentage points), versus 71% without the critic. A preliminary 15-dataset expert study yields Fleiss' kappa of 0.71 and 82% alignment with expert consensus. API cost is approximately USD 0.054 per dataset. These results support auditability and feasibility, while the limited benchmark, incomplete ablations, and single-model-family validation constrain claims about accuracy and generalization.

发表机构

• The University of Melbourne（墨尔本大学）

机构由 AI 辅助整理，请以论文原文为准。