Ayn：一个微小却具有竞争力的印度法律语言模型，从头开始预训练
Ayn: A Tiny yet Competitive Indian Legal Language Model Pretrained from Scratch
AI总结：
本文提出Ayn，一个预训练的印度法律领域小型语言模型，在法律案例判断预测任务中超越80倍大的LLM，在摘要任务中与30倍大的LLM竞争，并在通用任务中与大模型竞争。
摘要：
解码器-only的大语言模型（LLMs）目前是许多自然语言处理（NLP）应用的首选模型。通过指令微调和提示方法，此类LLMs已被高效地用于解决通用和领域特定的任务。然而，训练它们成本高昂，而且在某种程度上，使用它们同样昂贵，人们不禁会问LLMs是否可以被领域特定的微型语言模型（TLMs）取代，这些模型通常包含少于10亿参数。在本研究中，我们通过将一个在特定领域上从头开始预训练185个A100小时的8800万参数TLM与不同大小（1B到8B）的LLMs进行比较，来回答这个问题。我们特别展示，我们的法律TLM，Ayn，在法律案例判断预测任务中确实能够超越80倍大的LLMs，在摘要任务中与30倍大的LLMs竞争，并且在通用任务中仍能与这些更大的LLMs竞争。
英文摘要：
Decoder-only Large Language Models (LLMs) are currently the model of choice for many Natural Language Processing (NLP) applications. Through instruction fine-tuning and prompting approaches, such LLMs have been efficiently used to solve both general and domain-specific tasks. However, they are costly to train and, to a certain extent, costly to use as well, and one can wonder whether LLMs can be replaced by domain-specific Tiny Language Models (TLMs), which typically contain less than 100M parameters. We address this question in this study by comparing the performance of an 88M TLM pretrained from scratch for 185 A100 hours on a specific domain with a domain-specific tokenizer (here, the Indian legal domain) with LLMs of various sizes between 1B and 8B for solving domain-specific tasks. We show in particular that our legal TLM, Ayn, can indeed outperform LLMs up to 80 times larger on the legal case judgment prediction task, rival LLMs up to 30 times larger on the summarization task, and still be competitive with these larger LLMs on general tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
