Traditional methods for aligning Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on implicit principles, limiting interpretability. Constitutional AI (CAI) offers an explicit, rule-based framework for guiding LLM alignment. Building on this, we refine the Inverse Constitutional AI (ICAI) algorithm, which extracts constitutions from preference datasets. By improving principle generation, clustering, and embedding processes, our approach enhances the accuracy and generalizability of extracted principles across synthetic and real-world datasets. Our results highlight the potential of these principles to foster more transparent and adaptable alignment methods, offering a promising direction for future advancements beyond traditional fine-tuning.
Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学);University of California, Berkeley(加州大学伯克利分校);Potsdam Institute for Climate Impact Research(波茨坦气候影响研究所);Ghent University(根特大学);Allen Institute for AI(艾伦人工智能研究所);University of Maryland(马里兰大学);EleutherAI;Union College(联合学院);Istanbul Bilgi University(伊斯坦布尔比尔吉大学)
T-POP: Test-Time Personalization with Online Preference Feedback
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));East China Normal University(华东师范大学);Shenzhen Loop Area Institute(深圳河套学院);Tianjin University(天津大学);The Chinese University of Hong Kong(香港中文大学);Beihang University(北京航空航天大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州))
Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院通用人工智能国家重点实验室);Shanghai Jiao Tong University(上海交通大学);Zhongguancun Academy(中关村学院)
The Reward Model Selection Crisis in Personalized Alignment
School of Informatics, University of Edinburgh(信息学院,爱丁堡大学);Agency for Science, Technology and Research (A*STAR)(科技研究局);Samsung AI Research Center(三星人工智能研究中心)
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
Department of Statistics, University of Georgia(统计学系,佐治亚大学);School of Computing, University of Georgia(计算学院,佐治亚大学);Department of Biostatistics, Boston University(生物统计学系,波士顿大学);Department of Epidemiology & Biostatistics, University of Georgia(流行病学与生物统计学系,佐治亚大学);School of Computer and Cyber Sciences, Augusta University(计算机与网络科学学院,奥古斯塔大学);School of Electrical and Computer Engineering, University of Georgia(电气与计算机工程学院,佐治亚大学);Department of Statistics & Data Science, University of Arizona(统计学与数据科学系,亚利桑那大学);Kellogg School of Management, Northwestern University(凯洛格管理学院,西北大学);Department of Statistics, Harvard University(统计学系,哈佛大学);School of Computer Science, Carnegie Mellon University(计算机科学学院,卡内基梅隆大学)