As Autonomous Systems (AS) become more ubiquitous in society, more responsible for our safety and our interaction with them more frequent, it is essential that they are trustworthy. Assessing the trustworthiness of AS is a mandatory challenge for the verification and development community. This will require appropriate standards and suitable metrics that may serve to objectively and comparatively judge trustworthiness of AS across the broad range of current and future applications. The meta-expression `trustworthiness' is examined in the context of AS capturing the relevant qualities that comprise this term in the literature. Recent developments in standards and frameworks that support assurance of autonomous systems are reviewed. A list of key challenges are identified for the community and we present an outline of a process that can be used as a trustworthiness assessment framework for AS.
Collaborative Trustworthiness for Good Decision Making in Autonomous Systems
Institute of Computer and Network Engineering (IDA) Technische Universität Braunschweig（计算机与网络工程学院（IDA） 拜鲁恩堡大学）；Chair of Embedded Systems TU Dortmund University（嵌入式系统教授 鄂尔敦大学）；Robert Bosch Corporate Research（博世企业研究）
Goethe University Frankfurt（法兰克福大学）；Intel Labs（英特尔实验室）；Arcada University of Applied Sciences（阿卡达应用科学大学）；Seoul National University（首尔国立大学）
Assessing the trustworthiness of artificial intelligence systems requires knowledge from many different disciplines. These disciplines do not necessarily share concepts between them and might use words with different meanings, or even use the same words differently. Additionally, experts from different disciplines might not be aware of specialized terms readily used in other disciplines. Therefore, a core challenge of the assessment process is to identify when experts from different disciplines talk about the same problem but use different terminologies. In other words, the problem is to group…
University of Bristol（布里斯托大学）；Open University（开放大学）；Imperial College London（伦敦帝国理工学院）；Lancaster University（兰卡斯特大学）；University of Edinburgh（爱丁堡大学）；Bauhaus University Weimar（魏玛包豪斯大学）
Heriot-Watt University（赫瑞-瓦特大学）；City, University of London（伦敦大学城市学院）
There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents, road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing - knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), w…
Autonomous systems with cognitive features are on their way into the market. Within complex environments, they promise to implement complex and goal oriented behavior even in a safety related context. This behavior is based on a certain level of situational awareness (perception) and advanced de-cision making (deliberation). These systems in many cases are driven by artificial intelligence (e.g. neural networks). The problem with such complex systems and with using AI technology is that there is no generally accepted approach to ensure trustworthiness. This paper presents a framework to exactl…
Heriot-Watt University（赫瑞瓦特大学）；City, University of London（伦敦大学城市学院）
Context: Demonstrating high reliability and safety for safety-critical systems (SCSs) remains a hard problem. Diverse evidence needs to be combined in a rigorous way: in particular, results of operational testing with other evidence from design and verification. Growing use of machine learning in SCSs, by precluding most established methods for gaining assurance, makes operational testing even more important for supporting safety and reliability claims. Objective: We use Autonomous Vehicles (AVs) as a current example to revisit the problem of demonstrating high reliability. AVs are making thei…
Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarios
Zhongguancun Laboratory（中关村实验室）；Beihang University（北京航空航天大学）；Institute of Data Space, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；A*STAR（新加坡科技研究局）
"A Good Bot Always Knows Its Limitations": Assessing Autonomous System Decision-making Competencies through Factorized Machine Self-confidence
RTX Technology Research Center（RTX技术研究中心）；Smead Aerospace Engineering Sciences, University of Colorado Boulder（科罗拉多大学博尔德分校斯米德航空航天科学学院）；Allen Institute for Brain Science（艾伦脑科学研究所）
A new generation of increasingly autonomous and self-learning embodied systems is about to be developed. When deploying embodied systems into a real-life context we face various engineering challenges, as it is crucial to coordinate the behavior of embodied systems in a beneficial manner, ensure their compatibility with our human-centered social values, and design verifiably safe and reliable human-machine interaction. We are arguing that traditional systems engineering is coming to a climacteric from embedded to embodied systems, and with assuring the trustworthiness of dynamic federations of…
Assessing Drivers' Situation Awareness in Semi-Autonomous Vehicles: ASP based Characterisations of Driving Dynamics for Modelling Scene Interpretation and Projection
German Aerospace Center (DLR)（德国航空航天中心）；Institute for Systems Engineering for Future Mobility（未来出行系统工程研究所）
Engineering Trustworthy Agentic AI for Critical Systems
Texas A&M University（德克萨斯A&M大学）；Innovations in Systems Trust and Resilience (iSTAR) Laboratory（系统信任与弹性创新（iSTAR）实验室）；Texas A&M Institute of Data Science - Security, Privacy and Resilience for Trusted AI (SPARTA) Thematic Lab.（德克萨斯A&M大学数据科学研究所 - 可信人工智能的安全、隐私与弹性（SPARTA）主题实验室）；Texas A&M Global Cyber Research Institute (GCRI)（德克萨斯A&M大学全球网络研究所）
Loughborough University（拉夫堡大学）；University of Manchester（曼彻斯特大学）；Ulster University（阿尔斯特大学）
The trustworthiness of Robots and Autonomous Systems (RAS) has gained a prominent position on many research agendas towards fully autonomous systems. This research systematically explores, for the first time, the key facets of human-centered AI (HAI) for trustworthy RAS. In this article, five key properties of a trustworthy RAS initially have been identified. RAS must be (i) safe in any uncertain and dynamic surrounding environments; (ii) secure, thus protecting itself from any cyber-threats; (iii) healthy with fault tolerance; (iv) trusted and easy to use to allow effective human-machine inte…
TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving
International Centre for Theoretical Sciences（国际理论科学中心）；Tata Institute of Fundamental Research（塔塔基础研究所）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Kavli IPMU (WPI), The University of Tokyo（东京大学 Kavli 宇宙物理学与数学起源研究所（WPI））；Harish-Chandra Research Institute（哈里什-钱德拉研究所）；Homi Bhabha National Institute（霍米·巴巴国立研究所）；Duke University（杜克大学）；University of Pennsylvania（宾夕法尼亚大学）；Indian Institute of Technology Madras（马德拉斯印度理工学院）；Harvard University（哈佛大学）；Jawaharlal Nehru Centre for Advanced Scientific Research（贾瓦哈拉尔·尼赫鲁高级科学研究中心）；Chennai Mathematical Institute（金奈数学研究所）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
University of Oxford（牛津大学）；University of Nottingham（诺丁汉大学）
The automotive industry has witnessed an increasing level of development in the past decades; from manufacturing manually operated vehicles to manufacturing vehicles with a high level of automation. With the recent developments in Artificial Intelligence (AI), automotive companies now employ blackbox AI models to enable vehicles to perceive their environments and make driving decisions with little or no input from a human. With the hope to deploy autonomous vehicles (AV) on a commercial scale, the acceptance of AV by society becomes paramount and may largely depend on their degree of transpare…
Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems
Department of Computer Science Princeton University（计算机科学系普林斯顿大学）；Department of Computer Science Department of Psychology Princeton University（计算机科学系心理学系普林斯顿大学）
