arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.16633cs.CLq-fin.GN
收藏

零并非英雄:面向金融任务的 LLM 零样本性能基准评测

Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks

Georgia Institute of Technology(佐治亚理工学院)

Agam Shah, Sudheer Chava

AI总结:

本文基准评测了 ChatGPT 与开源生成式 LLM 在金融任务上的零样本性能,发现其表现良好但通常不及微调 RoBERTa,并指出生成式模型标注耗时。

摘要:

近来,像 ChatGPT 这样的大语言模型(LLMs)在许多自然语言处理任务上展现出令人印象深刻的零样本性能。本文考察零样本 LLM 在金融领域的有效性。我们将 ChatGPT 及一些开源生成式 LLM 的零样本表现,与在标注数据上微调的 RoBERTa 进行比较。我们围绕数据标注、性能差距以及在金融领域采用生成式模型的可行性,处理三个相互关联的研究问题。我们的发现表明,即使没有标注数据,ChatGPT 也表现良好,但微调模型通常仍优于它。我们的研究还强调,使用生成式模型进行标注可能非常耗时。我们的代码库已在 GitHub 上以 CC BY-NC 4.0 许可公开。

英文摘要:

Recently large language models (LLMs) like ChatGPT have shown impressive performance on many natural language processing tasks with zero-shot. In this paper, we investigate the effectiveness of zero-shot LLMs in the financial domain. We compare the performance of ChatGPT along with some open-source generative LLMs in zero-shot mode with RoBERTa fine-tuned on annotated data. We address three inter-related research questions on data annotation, performance gaps, and the feasibility of employing generative models in the finance domain. Our findings demonstrate that ChatGPT performs well even without labeled data but fine-tuned models generally outperform it. Our research also highlights how annotating with generative models can be time-intensive. Our codebase is publicly available on GitHub under CC BY-NC 4.0 license.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑