We find that the best publicly available LLMs like GPT-4 and Claude currently perform poorly on basic legal text handling. This motivates the creation of a benchmark consisting of examples that lawyers and paralegals would expect LLMs to handle zero-shot, such as looking up the text at a line of a witness deposition or at a subsection of a contract. LLMs' poor performance on this benchmark casts into doubt their reliability as-is for legal practice. However, fine-tuning on our training set brings even a small model to near-perfect performance. This benchmark will be useful for fine-tuning LLMs for downstream legal tasks, as well as for tracking LLMs' reliability as-is for basic legal tasks.
AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensics
Chair for Cybersecurity, University of Augsburg, Augsburg, Germany（网络安全系，乌尔姆大学，乌尔姆，德国）
IPBench: Benchmarking the Knowledge of Large Language Models in Intellectual Property
Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China（深圳高性能数据挖掘重点实验室，深圳先进技术研究院，中国科学院，中国）；University of Chinese Academy of Sciences, China（中国科学院大学，中国）；Dalian University of Technology, China（大连理工大学，中国）；School of Biomedical Engineering, UNSW Sydney, Australia（生物医学工程学院，新南威尔士大学悉尼分校，澳大利亚）；Shenzhen MSU-BIT University, China（深圳MSU-BIT大学，中国）；Shenzhen University of Advanced Technology, China（深圳大学先进技术学院，中国）
CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
首字母缩略词和长形式在研究文件中很常见，在科学和法律领域的文件中更是如此。这些文档中使用的许多首字母缩略词都是特定于领域的，很少在普通文本语料库中找到。因此，基于transformer的NLP模型通常会检测首字母缩略词标记的OOV（Out of Vocabulary），特别是对于非英语语言，并且在提取过程中将首字母缩略词链接到其长形式时，其性能会受到影响。此外，像BERT这样的预训练的Transformer模型并不专门处理科学和法律文档。这些要点是这项工作背后的首要动机，我们提出了一个新的框架CABACE：字母缩写词提取的字符感知BERT，它考虑到文本中的字符序列，并适应于科学和法律领域的掩蔽语言建模。我们进一步使用具有增强的损失函数的目标，将最大损失和掩码损失项添加到用于训练CABACE的标准交叉熵损失。我们进一步利用伪标签和对抗性数据生成来提高框架的通用性。实验结果证明了所提出的框架相比，各种基线的优越性。此外，我们表明，所提出的框架比基线模型更适合zero-shot推广到非英语语言，从而加强了我们的方法的有效性。我们的团队BacKGProp在SDU AAAI-22的首字母缩略词提取（AE）共享任务的全球排行榜上获得了法语数据集的最高分，丹麦语和越南语的第二高分，以及英语法律数据集的第三高分。
Combining topic modelling and citation network analysis to study case law from the European Court on Human Rights on the right to respect for private and family life
