基于旅行聊天机器人用例的大语言模型微调方法与评估指标比较
A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case
AI总结：
本研究以旅行聊天机器人为用例，比较QLoRA、RAFT和RLHF等LLM微调方法及多种评估指标，发现GPT-4评估最接近人工评估，Mistral RAFT经RLHF后性能最佳。
摘要：
本研究比较了大语言模型（LLM）微调方法，包括量化低秩适配器（QLoRA）、检索增强微调（RAFT）和基于人类反馈的强化学习（RLHF），并进一步比较了LLM评估方法，包括“黄金答案”端到端（E2E）基准方法、传统自然语言处理（NLP）指标、RAG评估（Ragas）、OpenAI GPT-4评估指标和人工评估，使用旅行聊天机器人作为用例。旅行数据集来源于Reddit API，通过请求旅行相关子版块的帖子来获取旅行相关的对话提示和个性化旅行体验，并针对每种微调方法进行了数据增强。我们使用了两个预训练LLM用于微调研究：LLaMa 2 7B和Mistral 7B。QLoRA和RAFT被应用于这两个预训练模型。这些模型的推理结果根据上述指标进行了广泛评估。根据人工评估和部分GPT-4指标，最佳模型是Mistral RAFT，因此该模型经过了基于人类反馈的强化学习（RLHF）训练流程，最终被评估为最佳模型。我们的主要发现是：1）定量指标和Ragas指标与人工评估不一致；2）OpenAI GPT-4评估与人工评估最一致；3）在评估中保持人类参与至关重要，因为；4）传统NLP指标不足；5）Mistral总体上优于LLaMa；6）RAFT优于QLoRA，但仍需要后处理；7）RLHF显著提高了模型性能。后续步骤包括提高数据质量、增加数据数量、探索RAG方法，并将数据收集聚焦于特定城市，这将通过缩小关注范围来提高数据质量，同时创造有用的产品。
英文摘要：
This research compares large language model (LLM) fine-tuning methods, including Quantized Low Rank Adapter (QLoRA), Retrieval Augmented fine-tuning (RAFT), and Reinforcement Learning from Human Feedback (RLHF), and additionally compared LLM evaluation methods including End to End (E2E) benchmark method of "Golden Answers", traditional natural language processing (NLP) metrics, RAG Assessment (Ragas), OpenAI GPT-4 evaluation metrics, and human evaluation, using the travel chatbot use case. The travel dataset was sourced from the the Reddit API by requesting posts from travel-related subreddits to get travel-related conversation prompts and personalized travel experiences, and augmented for each fine-tuning method. We used two pretrained LLMs utilized for fine-tuning research: LLaMa 2 7B, and Mistral 7B. QLoRA and RAFT are applied to the two pretrained models. The inferences from these models are extensively evaluated against the aforementioned metrics. The best model according to human evaluation and some GPT-4 metrics was Mistral RAFT, so this underwent a Reinforcement Learning from Human Feedback (RLHF) training pipeline, and ultimately was evaluated as the best model. Our main findings are that: 1) quantitative and Ragas metrics do not align with human evaluation, 2) Open AI GPT-4 evaluation most aligns with human evaluation, 3) it is essential to keep humans in the loop for evaluation because, 4) traditional NLP metrics insufficient, 5) Mistral generally outperformed LLaMa, 6) RAFT outperforms QLoRA, but still needs postprocessing, 7) RLHF improves model performance significantly. Next steps include improving data quality, increasing data quantity, exploring RAG methods, and focusing data collection on a specific city, which would improve data quality by narrowing the focus, while creating a useful product.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
