评估生成式语言模型在分类任务中的表现:环境与气候变化领域的性能与自我评估能力
Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain
AI总结:
本文比较了GPT3.5、Llama2和Gemma与BERT基线在环境与气候变化分类任务上的性能,并分析其置信度校准能力,揭示生成式模型的优势与局限。
摘要:
本文考察了两个大语言模型(LLMs)GPT3.5和Llama2以及一个小语言模型(SLM)Gemma在气候变化(CC)和环境领域内三个不同分类任务上的表现。我们采用基于BERT的模型作为基线,比较它们与这些基于Transformer的模型的效能。此外,我们通过分析这些文本分类任务中言语化置信度分数的校准情况,评估了模型的自我评估能力。我们的研究结果表明,虽然基于BERT的模型通常优于LLMs和SLM,但大型生成式模型的表现仍然值得关注。此外,我们的校准分析显示,尽管Gemma在初始任务中校准良好,但之后产生不一致的结果;Llama的校准较为合理,而GPT始终表现出较强的校准性。通过这项研究,我们旨在为关于生成式语言模型在解决地球上一些最紧迫问题方面的实用性和有效性的持续讨论做出贡献,突出它们在生态学和气候变化背景下的优势与局限。
英文摘要:
This paper examines the performance of two Large Language Models (LLMs), GPT3.5 and Llama2 and one Small Language Model (SLM) Gemma, across three different classification tasks within the climate change (CC) and environmental domain. Employing BERT-based models as a baseline, we compare their efficacy against these transformer-based models. Additionally, we assess the models' self-evaluation capabilities by analyzing the calibration of verbalized confidence scores in these text classification tasks. Our findings reveal that while BERT-based models generally outperform both the LLMs and SLM, the performance of the large generative models is still noteworthy. Furthermore, our calibration analysis reveals that although Gemma is well-calibrated in initial tasks, it thereafter produces inconsistent results; Llama is reasonably calibrated, and GPT consistently exhibits strong calibration. Through this research, we aim to contribute to the ongoing discussion on the utility and effectiveness of generative LMs in addressing some of the planet's most urgent issues, highlighting their strengths and limitations in the context of ecology and CC.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。