利用合成数据探索多样且细粒度的指令遵循能力
Diverse and Fine-Grained Instruction-Following Ability Exploration with Synthetic Data
AI总结：
本文提出细粒度指令遵循评估数据集DINGO，基于130节点类别树并结合GPT-4与人工生成多样化指令，为LLMs提供更全面评估和改进方向。
摘要：
指令遵循对于大型语言模型（LLMs）支持多样化的用户请求尤为关键。尽管现有工作在使LLMs与人类偏好对齐方面取得了进展，但由于真实世界用户指令的复杂性和多样性，评估其指令遵循能力仍然是一个挑战。现有评估方法虽然关注通用技能，但存在两个主要缺陷，即缺乏细粒度的任务级评估以及依赖单一的指令表达方式。为了解决这些问题，本文引入了DINGO，一个细粒度且多样化的指令遵循评估数据集，具有两个主要优势：（1）DINGO基于人工标注的、细粒度的多层次类别树，包含源自真实世界用户请求的130个节点；（2）DINGO包含由GPT-4和人类专家共同生成的多样化指令。通过大量实验，我们证明DINGO不仅能为LLMs提供更具挑战性和更全面的评估，还能提供任务级细粒度方向以进一步改进LLMs。
英文摘要：
Instruction-following is particularly crucial for large language models (LLMs) to support diverse user requests. While existing work has made progress in aligning LLMs with human preferences, evaluating their capabilities on instruction following remains a challenge due to complexity and diversity of real-world user instructions. While existing evaluation methods focus on general skills, they suffer from two main shortcomings, i.e., lack of fine-grained task-level evaluation and reliance on singular instruction expression. To address these problems, this paper introduces DINGO, a fine-grained and diverse instruction-following evaluation dataset that has two main advantages: (1) DINGO is based on a manual annotated, fine-grained and multi-level category tree with 130 nodes derived from real-world user requests; (2) DINGO includes diverse instructions, generated by both GPT-4 and human experts. Through extensive experiments, we demonstrate that DINGO can not only provide more challenging and comprehensive evaluation for LLMs, but also provide task-level fine-grained directions to further improve LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
