Model Context Protocol (MCP) servers contain a collection of thousands of open-source standardized tools, linking LLMs to external systems; however, existing datasets and benchmarks lack realistic, human-like user queries, remaining a critical gap in evaluating the tool usage and ecosystems of MCP servers. Existing datasets often do contain tool descriptions but fail to represent how different users portray their requests, leading to poor generalization and inflated reliability of certain benchmarks. This paper introduces the first large-scale MCP dataset featuring diverse, high-quality diverse user queries generated specifically to match 2800 tools across 308 MCP servers, developing on the MCP Zero dataset. Each tool is paired with multiple unique user personas that we have generated, to capture varying levels of user intent ranging from precise task requests, and ambiguous, exploratory commands, reflecting the complexity of real-world interaction patterns.
CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
University of Oxford（牛津大学）；Tony Blair Institute for Global Change（托尼·布莱尔全球变化研究所）；Google DeepMind（谷歌DeepMind）；UK AI Security Institute（英国人工智能安全研究所）；Imperial College London（帝国理工学院）；Downing Street（唐宁街10号）
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）
Commonsense reasoning research has so far been limited to English. We aim to evaluate and improve popular multilingual language models (ML-LMs) to help advance commonsense reasoning (CSR) beyond English. We collect the Mickey Corpus, consisting of 561k sentences in 11 different languages, which can be used for analyzing and improving ML-LMs. We propose Mickey Probe, a language-agnostic probing task for fairly evaluating the common sense of popular ML-LMs across different languages. In addition, we also create two new datasets, X-CSQA and X-CODAH, by translating their English versions to 15 oth…
MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；State Key Laboratory of Complex & Critical Software Environment（复杂关键软件环境国家重点实验室）；National Key Laboratory of Parallel and Distributed Computing（并行与分布式计算国家重点实验室）
mcp-proto-okn: Natural-language access to open scientific knowledge graphs through the Model Context Protocol
San Diego Supercomputer Center, University of California San Diego（圣地亚哥超级计算机中心，加州大学圣地亚哥分校）；The Scripps Research Institute（斯克里普斯研究所）；Amentum, Space Biosciences Division, NASA Ames Research Center（Amentum空间生物学部门，NASA阿姆斯研究中心）；Mate Bioservices（Mate生物服务）；Renaissance Computing Institute, University of North Carolina at Chapel Hill（复兴计算研究所，北卡罗来纳大学夏洛特分校）；Weill Institute for Neurosciences, Department of Neurology, University of California San Francisco（Weill神经科学研究所，神经病学系，加州大学旧金山分校）
Bypassing Document Ingestion: An MCP Approach to Financial Q&A
University of Groningen（Groningen大学）；Goethe University Frankfurt（法兰克福歌德大学）；Martin Luther University Halle-Wittenberg（哈雷-维滕贝格马丁路德大学）；Halle Institute for Economic Research (IWH)（哈雷经济研究所）
PII-Bench: Evaluating Query-Aware Privacy Protection Systems
Institute of Fintech, Fudan University（复旦大学金融科技学院）；Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University（复旦大学数据科学上海重点实验室）；Laboratory of Data Analytics and Security, Fudan University（复旦大学数据安全分析实验室）
ShieldNet: Network-Level Guardrails against Emerging Supply-Chain Injections in Agentic Systems
UIUC（伊利诺伊大学厄巴纳-香槟分校）；University of Chicago（芝加哥大学）；University of Georgia（佐治亚大学）；United States Military Academy（美国军事学院）；eBay；Johns Hopkins University（约翰霍普金斯大学）；UCSB（加州大学圣塔芭芭拉分校）；Virtue AI
Although representational retrieval models based on Transformers have been able to make major advances in the past few years, and despite the widely accepted conventions and best-practices for testing such models, a $\textit{standardized}$ evaluation framework for testing them has not been developed. In this work, we formalize the best practices and conventions followed by researchers in the literature, paving the path for more standardized evaluations - and therefore more fair comparisons between the models. Our framework (1) embeds the documents and queries; (2) for each query-document pair,…
Evaluating Query Efficiency and Accuracy of Transfer Learning-based Model Extraction Attack in Federated Learning
Center for Secure & Intelligent Critical Systems, Old Dominion University, Virginia, USA（安全与智能关键系统中心，旧 Dominion 大学，弗吉尼亚州，美国）；School of Cybersecurity, Old Dominion University, Virginia, USA（网络安全学院，旧 Dominion 大学，弗吉尼亚州，美国）
University of Southern California（南加州大学）；Facebook AI Research（Facebook人工智能研究院）
Providing systems the ability to relate linguistic and visual content is one of the hallmarks of computer vision. Tasks such as text-based image retrieval and image captioning were designed to test this ability but come with evaluation measures that have a high variance or are difficult to interpret. We study an alternative task for systems that match text and images: given a text query, the system is asked to select the image that best matches the query from a pair of semantically similar images. The system's accuracy on this Binary Image SelectiON (BISON) task is interpretable, eliminates th…
Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）
Although query-based systems (QBS) have become one of the main solutions to share data anonymously, building QBSes that robustly protect the privacy of individuals contributing to the dataset is a hard problem. Theoretical solutions relying on differential privacy guarantees are difficult to implement correctly with reasonable accuracy, while ad-hoc solutions might contain unknown vulnerabilities. Evaluating the privacy provided by QBSes must thus be done by evaluating the accuracy of a wide range of privacy attacks. However, existing attacks require time and expertise to develop, need to be m…
StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs
Southern University of Science and Technology（南方科技大学）；Alibaba Group（阿里巴巴集团）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Hong Kong（香港大学）
Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions
Department of Intelligent Software, Sungkyunkwan University（智能软件系，成均馆大学）；Department of Computer Science and Engineering, Sungkyunkwan University（计算机科学与工程系，成均馆大学）
Evaluating LLMs on Sequential API Call Through Automated Test Generation
The University of Tokyo（东京大学）；Macau University of Science and Technology（澳门科技大学）；Shandong University（山东大学）；Hong Kong University of Science and Technology（香港科技大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Alberta（阿尔伯塔大学）
