Alignment is a standard procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e. hallucination). In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps:\ supervised fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that training the LLM on new knowledge or unfamiliar texts can encourage hallucination. This makes SFT less factual as it trains on human labeled data that may be novel to the LLM. Furthermore, reward functions used in standard RL can also encourage hallucination, because it guides the LLM to provide more helpful responses on a diverse set of instructions, often preferring longer and more detailed responses. Based on these observations, we propose factuality-aware alignment, comprised of factuality-aware SFT and factuality-aware RL through direct preference optimization. Experiments show that our proposed factuality-aware alignment guides LLMs to output more factual responses while maintaining instruction-following capability.
Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
Tsinghua University, Institute for Interdisciplinary Information Sciences（清华大学，交叉信息研究院）；Shanghai Qi Zhi Institute（上海期智研究院）；South China Normal University, School of Electronic Science and Engineering（华南师范大学，电子科学与工程学院）；Guangzhou Richstone Data Technologies Co., Ltd.（广州瑞驰数据技术有限公司）；LMU Munich & Munich Center for Machine Learning（慕尼黑大学 & 慕尼黑机器学习中心）
UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models
The Chinese University of Hong Kong（香港中文大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The University of Hong Kong（香港大学）；The Hong Kong Polytechnic University（香港理工大学）；Hong Kong University of Science and Technology, Guangzhou（香港科技大学（广州））；MoE Key Laboratory of High Confidence Software Technologies（教育部高可信软件技术重点实验室）
FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs
UCLA（美国大学洛杉矶分校）；CUHK（香港中文大学）；HKUST (GZ)（香港科技大学（广州））；HKUST（香港科技大学）；Tsinghua University（清华大学）；ECNU（华东师范大学）；NVIDIA（英伟达）；UCL（伦敦大学学院）；HKU（香港大学）；University of Cambridge（剑桥大学）
Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation
National Biomarker Centre, CRUK-MI, Univ. of Manchester（英国曼彻斯特大学CRUK-MI国家生物标志物中心）；Idiap Research Institute（Idiap研究所）；Inflamalps SA（因弗马尔普斯公司）；Department of Computer Science, Univ. of Manchester（英国曼彻斯特大学计算机科学系）
Expanding before Inferring: Enhancing Factuality in Large Language Models through Premature Layers Interpolation
Sun Yat-Sen University（中山大学）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）；The Hong Kong Polytechnic University（香港理工大学）；UNSW, Sydney, NSW 2052, Australia（新南威尔士大学）；Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳高性能数据挖掘重点实验室，深圳先进技术研究院，中国科学院）
Graph Alignment Topology as an Inductive Bias for Grounding Detection
Department of Pediatrics, University of Illinois College of Medicine Peoria（伊利诺伊大学皮奥里亚医学院儿科部）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机与数据科学学院）；Carle Illinois College of Medicine, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校卡莱医学院）
ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents
Multiverse Computing；Parque Cientifico y Tecnológico de Gipuzkoa（吉普斯夸科技园）；Centre for Social Innovation（社会创新中心）；Donostia International Physics Center（多诺斯蒂亚国际物理中心）；Ikerbasque Foundation for Science（伊克尔巴斯克科学基金会）
EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Taobao & Tmall Group（淘宝天猫集团）；Shandong University（山东大学）；Peking University（北京大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
Wroclaw University of Science and Technology (WUST)（沃拉布大学科学与技术学院）；National Research Institute (NASK)（国家研究 institute）；Institute of Computer Science, Polish Academy of Sciences (IPI PAN)（波兰科学院计算机科学研究所）
MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Zhongguancun Academy（中关村学院）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent（腾讯）
Universitat Politècnica de València（瓦伦西亚理工大学）；University of Trento（特伦托大学）；University of Sheffield（谢菲尔德大学）；Bonn-Aachen International Center for Information Technology, University of Bonn（波恩-亚琛信息科技国际中心，波恩大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；University of Pisa（比萨大学）；University of Milano-Bicocca（米兰-比可卡大学）；ValgrAI Valencian Graduate School and Research Network of Artificial Intelligence（瓦伦西亚人工智能研究生学校和研究网络）
PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignment
Emory University（埃默里大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Google（谷歌）；Stanford University（斯坦福大学）；University of Southern California（南加州大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新 generation 人工智能技术及交叉应用重点实验室（东南大学），教育部，中国）
Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs
School of Engineering, Westlake University, Hangzhou, China（西湖大学工程学院）；School of Cyberspace Security, Nanjing University of Science and Technology, Nanjing, China（南京理工大学网络安全学院）
