FB-Bench：评估大语言模型对人类反馈响应能力的细粒度多任务基准
FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback
AI总结：
本文提出FB-Bench，一个包含591个样本、覆盖多类任务、反馈与回复缺陷的中文细粒度多任务基准，用于评估LLM对真实多轮交互中人类反馈的响应能力。
摘要：
人类反馈在人类与大语言模型（LLM）的交互中至关重要。然而，现有研究主要集中于在单轮对话中对LLM进行基准测试。即使在为多轮对话设计的基准中，用户输入也往往相互独立，忽视了真实使用场景中人类反馈细微且复杂的特性。为填补这一研究空白，我们推出了FB-Bench，这是一个细粒度、多任务的基准，旨在评估LLM在中文真实使用场景下对人类反馈的响应能力。基于两类主要交互场景，FB-Bench包含591个精心整理的样本，涵盖8种任务类型、5种回复缺陷类型和9种反馈类型。我们对大量主流LLM进行了广泛评估，揭示了它们在不同交互场景下表现的显著差异。进一步分析表明，任务、人类反馈以及先前回复的缺陷也会显著影响LLM的响应能力。我们的研究结果强调了当前模型的优势与局限，为未来研究提供了宝贵见解和方向。代码和数据集可在https://github.com/PKU-Baichuan-MLSystemLab/FB-Bench获取。
英文摘要：
Human feedback is crucial in the interactions between humans and Large Language Models (LLMs). However, existing research primarily focuses on benchmarking LLMs in single-turn dialogues. Even in benchmarks designed for multi-turn dialogues, the user inputs are often independent, neglecting the nuanced and complex nature of human feedback within real-world usage scenarios. To fill this research gap, we introduce FB-Bench, a fine-grained, multi-task benchmark designed to evaluate LLMs' responsiveness to human feedback under real-world usage scenarios in Chinese. Drawing from the two main interaction scenarios, FB-Bench comprises 591 meticulously curated samples, encompassing eight task types, five deficiency types of response, and nine feedback types. We extensively evaluate a broad array of popular LLMs, revealing significant variations in their performance across different interaction scenarios. Further analysis indicates that task, human feedback, and deficiencies of previous responses can also significantly impact LLMs' responsiveness. Our findings underscore both the strengths and limitations of current models, providing valuable insights and directions for future research. Code and datasets are available at https://github.com/PKU-Baichuan-MLSystemLab/FB-Bench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
