Capture-the-Flag (CTF) competitions are increasingly becoming a testbed for evaluating AI capabilities at solving security tasks, due to the controlled environments and objective success criteria. Existing evaluations have focused on how successful AI is at solving CTF challenges in isolation from human CTF players. As AI usage increases in both academic and industrial settings, it is equally likely that human players may collaborate with AI agents to solve challenges. This possibility exposes a key knowledge gap: how do humans perceive AI CTF assistance; when assistance is provided, how do they collaborate and is it effective with respect to human performance; how do humans assisted by AI compare to the performance of fully autonomous AI agents on the same challenges. We address this gap with the first empirical study of AI assistance in a live, onsite CTF. In a study with 41 participants, we qualitatively study (i) how participants' perception, trust, and expectations shift before versus after hands-on AI use, and (ii) how participants collaborate with an instrumented AI agent. Moreover, we also (iii) benchmark four autonomous AI agents on the same fresh challenge set to compare outcomes with human teams and analyze agent trajectories. We find that, as the competition progresses, teams increasingly delegate larger subtasks to the AI, giving it more agency. Interestingly, CTF challenges solving rates are often constrained not by model's reasoning capabilities, but rather by the human players: ineffective prompting and poor context specification become the primary bottleneck. Remarkably, autonomous agents that self-direct their prompting and tool use bypass this bottleneck and outperform most human teams, coming in second overall in the competition. We conclude with implications for the future design of CTF challenges and for building effective human-in-the-loop AI systems for security.
A Neuro-Symbolic Multi-Agent Approach to Legal-Cybersecurity Knowledge Integration
Department of Control and Computer Engineering, Politecnico di Torino, Italy(控制与计算机工程系,托斯卡纳理工学院);Department of Computer Science, Università di Torino, Italy(计算机科学系,托斯卡纳大学);School of Computing and Communications, Lancaster University, UK(计算与通讯学院,兰卡斯特大学);School of Computing, George Mason University, USA(计算学院,乔治·马歇尔大学)
Poster: SpiderSim: Multi-Agent Driven Theoretical Cybersecurity Simulation for Industrial Digitalization
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Benefits and Pitfalls of Using Capture the Flag Games in University Courses
用于练习网络安全技能的Capture the Flag(CTF)游戏的概念在非正式教育环境和休闲比赛中广泛存在。然而,它在大学课程中使用得并不多。本文总结了我们的经验,从使用危险CTF游戏作为家庭作业在介绍本科课程。我们对描述学生在游戏中的行为和课程表现的数据进行了分析,揭示了CTF任务设计中应该解决的四个方面:评分,脚手架,剽窃和所用CTF平台的学习分析能力。本文通过分享我们的建议来解决这些问题。我们相信,这些建议对于考虑使用CTF游戏进行大学课程评估的网络安全教师和CTF游戏框架开发人员非常有用。
Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities
Department of Information Systems, University of Maryland, Baltimore County(信息系统系,马里兰大学巴尔的摩分校);Department of Computer Science and Engineering, University at Buffalo(计算机科学与工程系,布法罗大学);Department of Computer Science, University of Colorado Boulder(计算机科学系,科罗拉多大学博尔德分校);Laboratory for Cybersecurity Dynamics, Department of Computer Science, University of Colorado Colorado Springs(网络安全动力实验室,科罗拉多大学科罗拉多斯普林斯分校)
Explainable AI-based Intrusion Detection System for Industry 5.0: An Overview of the Literature, associated Challenges, the existing Solutions, and Potential Research Directions
Hamad bin Khalifa University(哈马德·本·哈利法大学);Qatar Computing Research Institute(卡塔尔计算研究所);Munster Technological University(芒斯特理工大学);Toronto Metropolitan University(多伦多都会大学)
ProvAgent: Threat Detection Based on Identity-Behavior Binding and Multi-Agent Collaborative Attack Investigation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);University of Chinese Academy of Sciences(中国科学院大学);Zhongguancun Laboratory(中关村实验室)