Large language models possess impressive capabilities in generating programs (e.g., Python) from natural language descriptions to execute robotic tasks. However, these generated programs often contain errors that violate externally given task specifications. Without an effective method to verify their correctness, the reliable deployment of language models in real-world systems is practically infeasible. We develop a method that converts generated robot programs into an automaton-based representation and verifies them against task-relevant safety specifications. We establish a theorem that any arbitrary combination of the verified programs will also satisfy the safety specifications. Hence, the method eliminates the need to verify complex programs composed of multiple simpler ones, reducing computation complexity. We then introduce an automated fine-tuning procedure that leverages verification outcomes for supervision. By applying the theorem, this procedure only requires training the model to generate safe sub-components, thereby improving training efficiency. Empirical results on robot applications show a 30 percent increase in the probability of generating specification-compliant programs, with training time reduced by half compared to fine-tuning on generating full programs.
A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation
University of Liverpool（利物浦大学）；Purple Mountain Laboratories（紫金山实验室）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；The University of Manchester（曼彻斯特大学）；KU Leuven（鲁汶大学）
Reachable Set Computation and Safety Verification for Neural Networks with ReLU Activations
Vanderbilt University（范德堡大学）
神经网络已被广泛用于解决复杂的现实问题。由于神经网络的复杂性、非线性和非凸性，对神经网络输出行为的形式安全保证将是其在安全关键系统中应用的关键，本文研究了一类由ReLU（Rectified Linear Unit）激活函数构成的神经网络的输出可达集计算和安全性验证问题.提出了一种逐层计算输出可达集的方法。计算公式化的形式，一组操作的多面体，它可以有效地应用与援助的多面体计算工具。基于输出可达集计算结果，可以通过检查由多面体的并集描述的不安全区域和输出可达集的交集来执行ReLU神经网络的安全性验证。一个随机生成的ReLU神经网络的数值例子表明本文开发的方法的有效性。
Improving Neural Network Verification through Spurious Region Guided Refinement
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Intelligent Software（智能软件研究所）；Zhejiang University（浙江大学）；Singapore Management University（新加坡管理大学）
Internalizing Safety Understanding in Large Reasoning Models via Verification
National University of Singapore（国立新加坡大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Convex-Analytic View of Query Selection for Safety Verification of Markov Decision Processes
The Institute of Statistical Mathematics（统计数学研究所）；Tohoku University（东北大学）；RIKEN AIP（理化学研究所人工智能中心）；CyberAgent；National Institute of Informatics（信息学国立研究所）
TriGuard: Testing Model Safety with Attribution Entropy, Verification, and Drift
Center for Data Science, New York University, New York, USA（数据科学中心，纽约大学，纽约，美国）；University of New Haven, Connecticut, USA（新罕布什尔大学，康涅狄格，美国）；Queens College, City University of New York, USA（皇后学院，纽约城市大学，美国）
A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
University of Liverpool（利物浦大学）；University of Oxford（牛津大学）；Lancaster University（兰卡斯特大学）；Defence Science and Technology Laboratory (Dstl)（国防科学技术实验室）；Queen’s University Belfast（贝尔法斯特女王大学）
