arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.15902cs.CLcs.AI
收藏

论预训练基础LLM在德国法律教育分析中的适用性

On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education

Universität Passau(帕绍大学);Institut für Rechtsdidaktik, Universität Passau(帕绍大学法律教学法研究所)

Lorenz Wendlinger, Christian Braun, Abdullah Al Zubaer, Simon Alexander Nonn, Sarah Großkopf, Christofer Fellicious, Michael Granitzer

AI总结:

本文评估了预训练基础LLM在德国法律教育分析中的适用性,发现其在特定任务和复杂语境下表现不佳,为此引入基于RAG的提示示例选择方法显著提升预测,并证实其在低标注数据场景下优于基线。

摘要:

我们表明,当前的开源基础LLM具备指令能力和德国法律背景知识,足以在教育语境下进行某些法律分析。然而,在非常特定的任务中,例如对“Gutachtenstil”鉴定式风格组件进行分类,或在复杂语境下(如完整的法律意见书),模型能力会崩溃。即使采用扩展上下文和有效的提示策略,它们也无法匹配Bag-of-Words基线。为应对此问题,我们引入了一种基于检索增强生成(RAG)的提示示例选择方法,在高数据可用性场景下显著改善了预测效果。我们进一步评估了预训练LLM在论证挖掘和自动作文评分两项标准任务上的表现,发现其更为胜任。总体而言,在缺乏或仅有少量标注数据的场景下,预训练LLM优于基线,且思维链提示在零样本情形下能进一步提供帮助。

英文摘要:

We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑