The rise of LLM-based agents shows great potential to revolutionize task planning, capturing significant attention. Given that these agents will be integrated into high-stake domains, ensuring their reliability and safety is crucial. This paper presents an Agent-Constitution-based agent framework, TrustAgent, with a particular focus on improving the LLM-based agent safety. The proposed framework ensures strict adherence to the Agent Constitution through three strategic components: pre-planning strategy which injects safety knowledge to the model before plan generation, in-planning strategy which enhances safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Our experimental results demonstrate that the proposed framework can effectively enhance an LLM agent's safety across multiple domains by identifying and mitigating potential dangers during the planning. Further analysis reveals that the framework not only improves safety but also enhances the helpfulness of the agent. Additionally, we highlight the importance of the LLM reasoning ability in adhering to the Constitution. This paper sheds light on how to ensure the safe integration of LLM-based agents into human-centric environments. Data and code are available at https://github.com/agiresearch/TrustAgent.
SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems
Texas A&M University（德州农工大学）；UC San Diego（加州大学圣地亚哥分校）；UC Irvine（加州大学欧文分校）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan（密歇根大学）；Columbia University（哥伦比亚大学）；Meta
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学工程学院、计算机科学与工程系）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP
Department of Computer Science and Engineering, BRAC University（布拉克大学计算机科学与工程系）；Department of Electrical and Electronic Engineering, Rajshahi University of Engineering and Technology（拉贾沙希工程与技术大学电子与电气工程系）
Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologies
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；MBZUAI（马克斯·普朗克人工智能研究所）；McGill University（麦吉尔大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）
Green Shielding: A User-Centric Approach Towards Trustworthy AI
University of California, Berkeley（加州大学伯克利分校）；University of Melbourne（墨尔本大学）；University of California, San Francisco（加州大学旧金山分校）；University of Georgia（佐治亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
TrajAD: Trajectory Anomaly Detection for Trustworthy LLM Agents
School of Software, Shandong University（山东大学软件学院）；Sonli Holding Group Co., Ltd.（山东利集团有限公司）；Shandong Huazhi Talent Technology Co., Ltd.（山东华智人才科技有限公司）；Information Technology Service Center of People’s Court（人民法院信息技术服务中心）
Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty
Manning College of Information and Computer Sciences, UMass Amherst, MA, USA（马萨诸塞大学阿姆赫斯特曼宁信息与计算机科学学院）；Center for Healthcare Organization and Implementation Research, VA Bedford Health Care（医疗组织与实施研究中心）；Miner School of Computer and Information Sciences, UMass Lowell, MA, USA（米纳尔计算机与信息科学学院）
Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey
Vanderbilt University（范德堡大学）；University of Notre Dame（圣母大学）；University of Oregon（俄勒冈大学）；Meta；Oracle Health AI（甲骨文健康人工智能公司）；Adobe Research（奥多比研究院）；Cisco AI Research（思科人工智能研究院）；North Carolina State University（北卡罗来纳州立大学）；The Hong Kong Polytechnic University（香港理工大学）；Air Force Research Lab（空军研究实验室）
