测量与缓解多语言刑事法庭中大语言模型的过度对齐
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
AI总结：
针对刑事法律场景中大语言模型因安全护栏导致拒绝响应（过度对齐）的问题，提出TF-RefusalBench基准，并评估提示工程与拒绝方向消融（abliteration）等缓解方法。
摘要：
尽管大语言模型在法律领域的广泛适用性因其可靠性和任何错误的严重性而备受争议，但具有充分理解和缓解风险的狭窄用途已经出现。值得注意的是，瑞士联邦最高法院使用小型本地模型进行四种官方语言的初步翻译和短段落摘要。然而，这种使用在刑法背景下具有挑战性。由于员工日常处理的裁决和案件可能包含暴力和性犯罪的详细描述，他们的合法工作因模型护栏（过度对齐）激活导致的拒绝和免责声明而受到损害。为了测量这一现象，我们引入了TF-RefusalBench，这是一个基于瑞士最高法院公开裁决的多语言刑事法律翻译和摘要基准。TF-RefusalBench包含法语、德语、意大利语和英语共5,200个提示，对应于常见任务提示和可能触发拒绝的段落。然后，我们使用TF-RefusalBench证明过度对齐是一个多方面的现象，受模型、提示和正在处理的文本语言的影响，并且其影响不能仅从过度拒绝的角度评估，因为免责声明对任务忠实度有影响。最后，我们评估了使本地大语言模型能够执行刑事法律任务的方法，证明虽然提示可能有效，但abliteration（拒绝方向消融）消除了拒绝，同时对任务性能影响最小。
英文摘要：
While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged. Notably the Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across the four official languages. However, such usage is challenging in the context of Criminal Law. Since rulings and cases employees work on routinely can contain detailed descriptions of violent and sexual offenses, their legitimate work is compromised by refusals and disclaimers due to the activation of model guardrails (over-alignment). To measure this phenomenon, we introduce TF-RefusalBench, a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. TF-RefusalBench contains 5,200 total prompts across French, German, Italian, and English, corresponding to common task prompts and passages likely to trigger refusal. We then use TF-RefusalBench to show that over-alignment is a multifaceted phenomenon, influenced by the model and the prompt and text languages being processed, and that its impact cannot be evaluated solely from an over-refusal perspective, given the disclaimer's impact on task faithfulness. Finally, we evaluate approaches to enable on-premises LLMs for Criminal Law Tasks, demonstrating that while prompting can be effective, abliteration (refusal directions ablation) eliminates refusal with minimal impact on task performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
