arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09659cs.CL
收藏

利用开源大型语言模型进行母语识别

Leveraging Open-Source Large Language Models for Native Language Identification

Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);CLTL(语言与技术计算实验室)

Yee Man Ng, Ilia Markov

AI总结:

本研究探索开源大型语言模型在母语识别任务中的潜力,发现开箱即用时性能不及闭源模型,但微调后可达到与商业模型相当的水平。

摘要:

母语识别(NLI)——即根据一个人用第二语言(L2)写作的内容来识别其母语(L1)的任务——在取证、市场营销和第二语言习得领域具有应用价值。历史上,严重依赖广泛特征工程的传统机器学习方法在此任务上表现优于基于转换器的语言模型。近期,闭源生成式大型语言模型(LLMs),如GPT-4,在零样本设置下展现了在NLI上的卓越性能,包括在开放集分类中的有前景的结果。然而,闭源LLMs存在诸多缺点,如高成本和训练数据的不透明性。本研究探讨了使用开源LLMs进行NLI的潜力。我们的结果表明,开源LLMs在开箱即用时无法达到闭源LLMs的准确率水平。然而,当在标记训练数据上进行微调后,开源LLMs可以实现与商业LLMs相当的性能。

英文摘要:

Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑