通过直接偏好对齐提升量化大语言模型的对话能力
Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment
AI总结：
针对量化大语言模型对话能力下降的问题，提出量化感知直接偏好优化（QDPO），对齐量化与全精度模型，在多种语言上超越现有技术，提升对话性能。
摘要：
大语言模型（LLM）的快速发展促进了其向对话式聊天机器人的转变，这些机器人能够把握上下文细微差别并生成相关句子，通过指令微调和基于人类反馈的强化学习（RLHF）等先进技术，紧密贴近人类价值观。然而，通过后训练量化（PTQ）等技术实现LLM所需的计算效率，带来了诸如token翻转等挑战，这可能损害聊天机器人的性能。为此，我们提出了一种新颖的偏好对齐方法——量化感知直接偏好优化（QDPO），该方法将量化LLM与其全精度对应模型对齐，从而提升对话能力。在多种语言的两个指令微调LLM上进行的评估表明，与既有的PTQ和知识蒸馏微调技术相比，QDPO在提升对话能力方面表现出更优性能，标志着在开发高效且有效的对话LLM方面迈出了重要一步。
英文摘要：
The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
