TrustLLM: Trustworthiness in Large Language Models
Lehigh University(利哈伊大学);University of Notre Dame(圣母大学);Illinois Institute of Technology(伊利诺伊理工大学);CISPA(CISPA亥姆霍兹信息安全研究中心);University of Cambridge(剑桥大学);Institut Polytechnique de Paris(巴黎综合理工学院);William & Mary(威廉玛丽学院);Texas A&M University(德克萨斯农工大学);University of Georgia(佐治亚大学);Samsung Research America(三星美国研究院);Stanford University(斯坦福大学);University of Oxford(牛津大学)
Large language models (LLMs), exemplified by ChatGPT, have gained considerable attention for their excellent natural language processing capabilities. Nonetheless, these LLMs present many challenges, particularly in the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs emerges as an important topic. This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions. Specifically, we first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we further establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics. We then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of over 30 datasets. Our findings firstly show that in general trustworthiness and utility (i.e., functional effectiveness) are positively related. Secondly, our observations reveal that proprietary LLMs generally outperform most open-source counterparts in terms of trustworthiness, raising concerns about the potential risks of widely accessible open-source LLMs. However, a few open-source LLMs come very close to proprietary ones. Thirdly, it is important to note that some LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent that they compromise their utility by mistakenly treating benign prompts as harmful and consequently not responding. Finally, we emphasize the importance of ensuring transparency not only in the models themselves but also in the technologies that underpin trustworthiness. Knowing the specific trustworthy technologies that have been employed is crucial for analyzing their effectiveness.
XTRUST: On the Multilingual Trustworthiness of Large Language Models
School of Artificial Intelligence, Jilin University, China(吉林大学人工智能学院);The Hong Kong University of Science and Technology (Guangzhou), China(香港科学与技术大学(广州));Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University, China(未来科学国际中心)
Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of Sydney(悉尼大学);Renmin University of China(中国人民大学);University of Chinese Academy of Sciences(中国科学院大学)
TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health
Renmin University of China, China(中国人民大学);Beijing University of Posts and Telecommunications, China(北京邮电大学);Hefei University of Technology, China(合肥工业大学)
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
Tsinghua University, Institute for Interdisciplinary Information Sciences(清华大学,交叉信息研究院);Shanghai Qi Zhi Institute(上海期智研究院);South China Normal University, School of Electronic Science and Engineering(华南师范大学,电子科学与工程学院);Guangzhou Richstone Data Technologies Co., Ltd.(广州瑞驰数据技术有限公司);LMU Munich & Munich Center for Machine Learning(慕尼黑大学 & 慕尼黑机器学习中心)
TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models
University of Alberta(阿尔伯塔大学);Mila - Quebec Artificial Intelligence Institute(魁北克人工智能研究所);The University of Tokyo(东京大学);Macau University of Science and Technology(澳门科学技术大学)
AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic
Center of Research Excellence in AI and Data Science(人工智能与数据科学卓越研究中心);King Abdulaziz University(阿卜杜勒阿齐兹国王大学);ASAS AI(ASAS人工智能公司);University College London(伦敦大学学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alexandria University(亚历山大大学)
Eliciting Trustworthiness Priors of Large Language Models via Economic Games
University of Hong Kong(香港大学);The University of Hong Kong(香港大学);Peking University(北京大学);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);Beijing Key Laboratory of Behavior and Mental Health, Peking University(北京大学行为与心理健康重点实验室);IDG/McGovern Institute for Brain Research, Peking University(北京大学脑科学研究院);Peking-Tsinghua Center for Life Sciences, Peking University(北京大学-清华大学生命科学中心);Key Laboratory of Machine Perception, Ministry of Education, China(教育部机器感知重点实验室)
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(神经语言处理与机器智能中心,中国科学院自动化研究所)
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
University of Texas at Austin(德克萨斯大学奥斯汀分校);Drexel University(德雷塞尔大学);MIT(麻省理工学院);UIUC(伊利诺伊大学厄巴纳-香槟分校);Duke University(杜克大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);Center for AI Safety(人工智能安全中心);University of California, Berkeley(加州大学伯克利分校);University of Chicago(芝加哥大学)
Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
Department of Computer Science and Technology, College of AI, Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University(计算机科学与技术系、人工智能学院、人工智能研究所、清华-博世联合机器学习中心、THBI实验室、BNRist中心、清华大学);Institute of Artificial Intelligence, Beihang University(人工智能研究院、北航);RealAI
Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
The Pennsylvania State University(宾夕法尼亚州立大学);University of Pennsylvania(宾夕法尼亚大学);UTHealth Houston(德克萨斯大学健康科学中心休斯顿分校);Amazon(亚马逊公司);Rensselaer Polytechnic Institute(伦斯勒理工学院)
Transparentize the Internal and External Knowledge Utilization in LLMs with Trustworthy Citation
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems Institute of Automation, Chinese Academy of Sciences(认知与决策智能复杂系统重点实验室,中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of Chinese Academy of Sciences(中国科学院大学);Unisound Al Technology Co,Ltd(Unisound人工智能技术有限公司);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);Center for AI Safety(AI安全中心);Microsoft Corporation(微软公司);The Chinese University of Hong Kong(香港中文大学)
Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control
State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室);Alibaba Cloud(阿里云);Hong Kong Baptist University(香港浸会大学);School of Software Technology, Zhejiang University(浙江大学软件学院);Zhiyuan Research Institute(智源研究院);Fabu Inc.(fabu公司)
A closer look at how large language models trust humans: patterns and biases
The Hebrew University Business School(希伯来大学商学院);The Federmann Center for the study of Rationality(费德曼理性研究中心);The Hebrew University of Jerusalem(特拉维夫希伯来大学)
EyeGPT: Ophthalmic Assistant with Large Language Models
The Hong Kong Polytechnic University(香港理工大学);Centre for Eye and Vision Research (CEVR)(眼科与视觉研究中心);Sun Yat-sen University(中山大学);Shanghai General Hospital(上海市第一人民医院);Shanghai Jiao Tong University(上海交通大学)