We examine one particular dimension of AI governance: how to monitor and audit AI-enabled products and services throughout the AI development lifecycle, from pre-deployment testing to post-deployment auditing. Combining principles from formal methods with SoTA machine learning, we propose techniques that enable AI-enabled product and service developers, as well as third party AI developers and evaluators, to perform offline auditing and online (runtime) monitoring of product-specific (temporally extended) behavioral constraints such as safety constraints, norms, rules and regulations with respect to black-box advanced AI systems, notably LLMs. We further provide practical techniques for predictive monitoring, such as sampling-based methods, and we introduce intervening monitors that act at runtime to preempt and potentially mitigate predicted violations. Experimental results show that by exploiting the formal syntax and semantics of Linear Temporal Logic (LTL), our proposed auditing and monitoring techniques are superior to LLM baseline methods in detecting violations of temporally extended behavioral constraints; with our approach, even small-model labelers match or exceed frontier LLM judges. Our predictive and intervening monitors significantly reduce the violation rates of LLM-based agents while largely preserving task performance. We further show through controlled experiments that LLMs' temporal reasoning shows a pronounced degradation in accuracy with increasing event distance, number of constraints, and number of propositions.
SymGPT: Auditing Smart Contracts via Combining Symbolic Execution with Large Language Models
The Pennsylvania State University（宾夕法尼亚州立大学）；University of Connecticut（康涅狄格大学）；University of California San Diego（加州大学圣地亚哥分校）；University of Oxford（牛津大学）；SKLP, Institute of Computing Technology, Chinese Academy of Sciences（SKLP，计算技术研究所，中国科学院）
Offscript: Automated Auditing of Instruction Adherence in LLMs
University of Washington Information School Seattle Washington USA；University of Washington\ G. Allen School of Computer Science \& Engineering Seattle Washington USA；University of Washington Information School；University of Washington\ G. Allen School of Computer Science \& Engineering
Governing AI Forgetting: Auditing for Machine Unlearning Compliance
School of Science and Engineering, Shenzhen Institute of Artificial Intelligence and Robotics for Society, The Chinese University of Hong Kong, Shenzhen, China（科学与工程学院，深圳人工智能与机器人社会研究所，香港中文大学（深圳））；Data Science and Analytics Thrust, Information Hub, Hong Kong University of Science and Technology (Guangzhou), China（数据科学与分析方向，信息中心，香港科学与技术大学（广州））；Internet of Things Thrust, Information Hub, Hong Kong University of Science and Technology (Guangzhou), China（物联网方向，信息中心，香港科学与技术大学（广州））
Beyond Pixels: A Vector-to-Graph Framework for Reliable Schematic Auditing
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（人工智能与数字经济广东实验室）；Guangdong Power Grid Co., Ltd.（广东电网公司）；Shanghai University（上海大学）；Carleton University（卡尔顿大学）
School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）
Robots of the future are going to exhibit increasingly human-like and super-human intelligence in a myriad of different tasks. They are also likely going to fail and be incompliant with human preferences in increasingly subtle ways. Towards the goal of achieving autonomous robots, the robot learning community has made rapid strides in applying machine learning techniques to train robots through data and interaction. This makes the study of how best to audit these algorithms for checking their compatibility with humans, pertinent and urgent. In this paper, we draw inspiration from the AI Safety…
CogManip: Benchmarking Manipulative Behavior in Multi-Turn Interactions with Large Language Model
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；BrainCog AI Lab, CASIA（CASIA脑认知人工智能实验室）；Gaoling School of AI, Renmin University of China（中国人民大学 Gallagher人工智能学院）；Beijing-AISI（北京人工智能研究所）；Beijing Key Laboratory of Safe AI and Superalignment（北京安全人工智能与超对齐重点实验室）；School of Artificial Intelligence, UCAS（中国科学技术大学人工智能学院）；Huawei Technologies Co., Ltd.（华为技术有限公司）
From Code to Compliance: Assessing ChatGPT's Utility in Designing an Accessible Webpage -- A Case Study
Department of Computer Science, Norwegian University of Science and Technology（计算机科学系，挪威科学技术大学）；Department of Design, Norwegian University of Science and Technology（设计系，挪威科学技术大学）
Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplaces
School of Mechanical, Aerospace, and Manufacturing Engineering（机械、航空航天与制造工程学院）；University of Connecticut（康涅狄格大学）；Department of Computer Science（计算机科学系）；University of California Irvine（加州大学欧文分校）
When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following
Marquette University（马凯特大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Georgia Tech（佐治亚理工学院）；Cornell University（康奈尔大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Hikvision（海康威视）
Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight
University of Notre Dame（诺丁汉大学）；University of Michigan（密歇根大学）；Northeastern University（东北大学）；University of Washington（华盛顿大学）；Johns Hopkins University（约翰霍普金斯大学）
