Alignment with human preference prevents large language models (LLMs) from generating misleading or toxic content while requiring high-cost human feedback. Assuming resources of human annotation are limited, there are two different ways of allocating considered: more diverse PROMPTS or more diverse RESPONSES to be labeled. Nonetheless, a straightforward comparison between their impact is absent. In this work, we first control the diversity of both sides according to the number of samples for fine-tuning, which can directly reflect their influence. We find that instead of numerous prompts, more responses but fewer prompts better trigger LLMs for human alignment. Additionally, the concept of diversity for prompts can be more complex than responses that are typically quantified by single digits. Consequently, a new formulation of prompt diversity is proposed, further implying a linear correlation with the final performance of LLMs after fine-tuning. We also leverage it on data augmentation and conduct experiments to show its effect on different algorithms.
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Microsoft Research(微软研究院);University of Michigan(密歇根大学);Shanghai Jiao Tong University(上海交通大学);CUHKSZ(香港中文大学(深圳));THU(清华大学)
One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity
School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究学院);Department of Psychology, Harvard University(哈佛大学心理学系)
Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models
School of Computing University of Kent(肯特大学计算机学院);Leiden Institute of Advanced Computer Science Universiteit Leiden(莱顿先进计算机科学研究所);Cheriton School of Computer Science University of Waterloo(滑铁卢大学查里顿计算机科学学院)
Cross-linguistic disagreement as a conflict of semantic alignment norms in multilingual AI~Linguistic Diversity as a Problem for Philosophy, Cognitive Science, and AI~
Japan Advanced Institute of Science and Technology(日本先进科学技术学院);Victoria University of Wellington(惠灵顿维多利亚大学);Macquarie University(麦考瑞大学);Nanzan University(南山大学);Kyoto University(京都大学)
Not All Layers Need Tuning: Selective Layer Restoration Recovers Diversity
Department of Computer Science, National University of Singapore, Singapore(新加坡国立大学计算机科学系);Smart Systems Institute, National University of Singapore, Singapore, Singapore(新加坡国立大学智能系统研究所)
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
Wroclaw University of Science and Technology (WUST)(沃拉布大学科学与技术学院);National Research Institute (NASK)(国家研究 institute);Institute of Computer Science, Polish Academy of Sciences (IPI PAN)(波兰科学院计算机科学研究所)
Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Institute of Trustworthy Embodied Artificial Intelligence, Fudan University(复旦大学可信具身人工智能研究所);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心)
AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));State University of New York at Buffalo(纽约州立大学布法罗分校);Huawei International, Singapore(新加坡华为国际)
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Department of Linguistics and Translation, City University of Hong Kong(香港城市大学语言学及翻译学系)
DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment
Southern University of Science and Technology(南方科技大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Science and Technology of China(中国科学技术大学);University of Copenhagen(哥本哈根大学)
SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selection
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科学与技术大学计算机科学与工程系);Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学计算机科学与工程系);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);Platform and Content Group, Tencent(腾讯平台与内容组);Chinese Medicine Guangdong Laboratory(广东中医实验室)
Preserving Diversity in Supervised Fine-Tuning of Large Language Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Nanjing University(南京大学);Hong Kong University of Science and Technology(香港科技大学);University of Pennsylvania(宾夕法尼亚大学)