LeCoDe：用于交互式法律咨询对话评估的基准数据集
LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation
AI总结：
针对法律咨询交互性不足的问题，构建含3696段真实多轮对话的LeCoDe基准，提出双维度12指标评估框架，实验显示GPT-4等模型性能有限，并探索了增强策略。
摘要：
法律咨询对于保障个人权利和确保司法可及性至关重要，然而由于专业人员短缺，法律咨询成本高昂且许多人难以获得。尽管大型语言模型（LLMs）的最新进展为可扩展、低成本的法律援助提供了有前景的途径，但当前系统在处理真实世界咨询的交互性和知识密集型特性方面仍显不足。为应对这些挑战，我们引入了LeCoDe，一个包含3,696段法律咨询对话、共计110,008个对话轮次的真实世界多轮基准数据集，旨在评估和提升LLMs的法律咨询能力。借助LeCoDe，我们创新性地从短视频平台收集直播咨询，提供了真实的多轮法律咨询对话。法律专家的严格标注进一步增强了数据集的专业见解和专业知识。此外，我们提出了一个综合评估框架，从（1）澄清能力和（2）专业建议质量两个方面评估LLMs的咨询能力。该统一框架包含两个维度下的12项指标。通过对各种通用和领域特定LLMs的大量实验，我们的结果揭示了该任务的重大挑战，即使是像GPT-4这样的最先进模型，其澄清能力的召回率也仅为39.8%，建议质量的总体得分为59%，凸显了专业咨询场景的复杂性。基于这些发现，我们进一步探索了几种增强LLMs法律咨询能力的策略。我们的基准数据集有助于推进法律领域对话系统的研究，特别是在模拟更真实的用户-专家交互方面。
英文摘要：
Legal consultation is essential for safeguarding individual rights and ensuring access to justice, yet remains costly and inaccessible to many individuals due to the shortage of professionals. While recent advances in Large Language Models (LLMs) offer a promising path toward scalable, low-cost legal assistance, current systems fall short in handling the interactive and knowledge-intensive nature of real-world consultations. To address these challenges, we introduce LeCoDe, a real-world multi-turn benchmark dataset comprising 3,696 legal consultation dialogues with 110,008 dialogue turns, designed to evaluate and improve LLMs' legal consultation capability. With LeCoDe, we innovatively collect live-streamed consultations from short-video platforms, providing authentic multi-turn legal consultation dialogues. The rigorous annotation by legal experts further enhances the dataset with professional insights and expertise. Furthermore, we propose a comprehensive evaluation framework that assesses LLMs' consultation capabilities in terms of (1) clarification capability and (2) professional advice quality. This unified framework incorporates 12 metrics across two dimensions. Through extensive experiments on various general and domain-specific LLMs, our results reveal significant challenges in this task, with even state-of-the-art models like GPT-4 achieving only 39.8% recall for clarification and 59% overall score for advice quality, highlighting the complexity of professional consultation scenarios. Based on these findings, we further explore several strategies to enhance LLMs' legal consultation abilities. Our benchmark contributes to advancing research in legal domain dialogue systems, particularly in simulating more real-world user-expert interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
