arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.06166cs.CL
收藏

评估超参数如何影响大型语言模型的讽刺检测性能

Assessing how hyperparameters impact Large Language Models' sarcasm detection performance

Toronto Metropolitan University(多伦多都会大学)

Montgomery Gole, Andriy Miranskyy

AI总结:

本研究探索超参数对GPT和Llama-2模型讽刺检测性能的影响,在SARC2.0数据集上评估不同规模、版本的微调与零样本模型,发现同家族模型微调性能随规模单调提升,超参数调优也有影响,Llama-2-13b微调后达0.83的准确率和F1分数,与人类平均表现相当。

摘要:

讽刺检测对人类和机器而言均具挑战性。鉴于OpenAI的GPT和Meta的Llama-2模型强大的自然语言理解能力及高人气,本研究探索模型特性如何影响其讽刺检测表现。我们评估了不同规模、版本和超参数下的微调模型与零样本模型,实验在热门Self-Annotated Reddit Corpus(SARC2.0)讽刺数据集的政治平衡(pol-bal)部分开展。同模型家族内,微调性能随模型规模单调提升,超参数调优也影响性能。微调场景中,全精度Llama-2-13b达到0.83的最先进准确率和F1分数,与人类平均表现相当;零样本场景下,一个GPT-4模型取得具竞争力的结果,准确率0.70、F1分数0.75。此外,模型性能可能随版本更新升降,凸显每次版本更新后需重新评估性能的必要性。

英文摘要:

Sarcasm detection is challenging for both humans and machines. This work explores how model characteristics impact sarcasm detection in OpenAI's GPT, and Meta's Llama-2 models, given their strong natural language understanding, and popularity. We evaluate fine-tuned and zero-shot models across various sizes, releases, and hyperparameters. Experiments were conducted on the political and balanced (pol-bal) portion of the popular Self-Annotated Reddit Corpus (SARC2.0) sarcasm dataset. Fine-tuned performance improves monotonically with model size within a model family, while hyperparameter tuning also impacts performance. In the fine-tuning scenario, full precision Llama-2-13b achieves state-of-the-art accuracy and $F_1$-score, both measured at 0.83, comparable to average human performance. In the zero-shot setting, one GPT-4 model achieves competitive performance to prior attempts, yielding an accuracy of 0.70 and an $F_1$-score of 0.75. Furthermore, a model's performance may increase or decline with each release, highlighting the need to reassess performance after each release.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑