Large language models (LLMs) have shown tremendous success in following user instructions and generating helpful responses. Nevertheless, their robustness is still far from optimal, as they may generate significantly inconsistent responses due to minor changes in the verbalized instructions. Recent literature has explored this inconsistency issue, highlighting the importance of continued improvement in the robustness of response generation. However, systematic analysis and solutions are still lacking. In this paper, we quantitatively define the inconsistency problem and propose a two-stage training framework consisting of instruction-augmented supervised fine-tuning and consistency alignment training. The first stage helps a model generalize on following instructions via similar instruction augmentations. In the second stage, we improve the diversity and help the model understand which responses are more aligned with human expectations by differentiating subtle differences in similar responses. The training process is accomplished by self-rewards inferred from the trained model at the first stage without referring to external human preference resources. We conduct extensive experiments on recent publicly available LLMs on instruction-following tasks and demonstrate the effectiveness of our training framework.
CM-Align: Consistency-based Multilingual Alignment for Large Language Models
Key Laboratory of Big Data & Artificial Intelligence in Transportation, Beijing Jiaotong University, Ministry of Education(大数据与人工智能交通运输 key laboratory,北京交通大学,教育部);School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China(计算机科学与技术学院,北京交通大学,北京,中国);Pattern Recognition Center, WeChat AI, Tencent Inc, China(模式识别中心,微信AI,腾讯公司,中国)
Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains
School of Information Engineering, Minzu University of China(中国民族大学信息工程学院);National Language Resource Monitoring and Research Center of Minority Languages(少数民族语言资源监测与研究中心)
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);Institute of Automation, CAS(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Cross-linguistic disagreement as a conflict of semantic alignment norms in multilingual AI~Linguistic Diversity as a Problem for Philosophy, Cognitive Science, and AI~
Japan Advanced Institute of Science and Technology(日本先进科学技术学院);Victoria University of Wellington(惠灵顿维多利亚大学);Macquarie University(麦考瑞大学);Nanzan University(南山大学);Kyoto University(京都大学)
Quantifying Logical Consistency in Transformers via Query-Key Alignment
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究所);Moscow Institute of Physics and Technology(莫斯科物理技术学院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学);Artificial Intelligence Research Institute (AIRI)(人工智能研究所(AIRI))
On the Entity-Level Alignment in Crosslingual Consistency
Center for Information and Language Processing, LMU Munich(信息与语言处理中心,慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML));Sorbonne Université, CNRS, ISIR, France(索邦大学,CNRS,ISIR,法国)
Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness
Seoul National University(首尔国立大学)
我们探讨的任务,提高人物角色的一致性的对话代理。最近处理一致性的模型通常使用额外的自然语言推理(NLI)标签进行训练,或者将训练过的额外模块附加到生成代理以保持一致性。然而,这种额外的标签和培训可能是苛刻的。此外,我们发现,即使是表现最好的基于角色的代理是不敏感的矛盾的话。受社会认知和语用学的启发,我们通过一个假想的听众赋予现有的对话主体公共自我意识。我们的方法基于理性言语行为框架(Frank and Goodman,2012),可以强制对话代理避免说出矛盾。我们通过学习分心物选择来进一步扩展框架,这通常是手动或随机完成的。关于对话NLI的结果(Welleck等人,2019)和PersonaChat(Zhang et al.,2018)数据集表明,我们的方法减少了矛盾,提高了现有对话模型的一致性。此外,我们表明,它可以推广到提高上下文一致性超越人物角色的对话。
Beijing Academy of Artificial Intelligence(北京人工智能研究院);National University of Singapore(新加坡国立大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)