Lawma：专业化在法律标注中的力量
Lawma: The Power of Specialization for Legal Annotation
AI总结：
本研究提出CaselawQA基准，证明小型微调模型在法律标注任务上优于商业模型，为实证法律研究提供了经济高效的替代方案。
摘要：
法律文本的标注与分类是实证法律研究的核心组成部分。传统上，这些任务通常交由受过训练的研究助理完成。受语言建模进展的推动，实证法律学者越来越多地转向使用商业模型进行提示（prompting），希望这能减轻人工标注的巨大成本。尽管使用日益增多，我们对如何最佳利用大型语言模型进行法律标注的理解仍然有限。为弥合这一差距，我们引入了CaselawQA，一个包含260个法律标注任务的基准，其中几乎所有任务对机器学习社区而言都是全新的。我们证明，商业模型（如GPT-4.5和Claude 3.7 Sonnet）实现了非平凡但高度可变的准确率，通常达不到法律工作所需的性能水平。随后，我们证明小型、轻度微调的模型优于商业模型。通常，几百到一千个带标签的示例就足以实现更高的准确率。我们的工作为当前主导性的商业模型提示实践提供了一种可行的替代方案。对于具有一些可用标注数据的具体法律标注任务，研究人员很可能更适合使用微调后的开源模型。
英文摘要：
Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal scholars are increasingly turning to prompting commercial models, hoping that it will alleviate the significant cost of human annotation. Despite growing use, our understanding of how to best utilize large language models for legal annotation remains limited. To bridge this gap, we introduce CaselawQA, a benchmark comprising 260 legal annotation tasks, nearly all new to the machine learning community. We demonstrate that commercial models, such as GPT-4.5 and Claude 3.7 Sonnet, achieve non-trivial yet highly variable accuracy, generally falling short of the performance required for legal work. We then demonstrate that small, lightly fine-tuned models outperform commercial models. A few hundred to a thousand labeled examples are usually enough to achieve higher accuracy. Our work points to a viable alternative to the predominant practice of prompting commercial models. For concrete legal annotation tasks with some available labeled data, researchers are likely better off using a fine-tuned open-source model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
