arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.07162cs.CLcs.SE
收藏

LLM从应用评论中提取特征-情感对的效果如何?

How Effectively Do LLMs Extract Feature-Sentiment Pairs from App Reviews?

University of Tartu(塔尔图大学)

Faiz Ali Shah, Ahmed Sabir, Rajesh Sharma, Dietmar Pfahl

AI总结:

本研究系统评估了GPT-4、ChatGPT和Llama-2等LLM在零样本、1样本和5样本场景下从应用评论中提取特征-情感对的能力,发现GPT-4在零样本下比规则方法SAFE提升17% f1,但微调模型RE-BERT仍具优势,为LLM在该任务上的应用提供了客观基准。

摘要:

对用户评论进行自动分析以理解用户对应用功能(即应用特性)的情感,有助于使开发工作与用户期望和需求保持一致。大型语言模型(LLMs)如ChatGPT近期在多个新任务上展现了出色性能,且无需更新模型参数,即使用零样本或少量标注示例,但LLMs在特征特定情感分析方面的能力尚未被探索。本研究的目标是探索LLMs执行用户评论特征特定情感分析的能力。本研究比较了最先进的LLMs,包括GPT-4、ChatGPT和Llama-2 chat的不同变体,与先前提取应用特征及相关情感的方法在零样本、1样本和5样本场景下的性能。结果表明,在零样本场景下,GPT-4在提取应用特征的f1分数上比基于规则的SAFE高出17%,5样本进一步将其提升6%。然而,微调后的RE-BERT在f1分数上超过GPT-4 6%。在预测正面和中性情感方面,GPT-4在零样本设置下分别取得76%和45%的f1分数,在5样本设置下分别提升7%和23%。本研究对专有和开源LLMs进行了全面评估,以客观评估它们在提取特征-情感对方面的性能。

英文摘要:

Automatic analysis of user reviews to understand user sentiments toward app functionality (i.e. app features) helps align development efforts with user expectations and needs. Recent advances in Large Language Models (LLMs) such as ChatGPT have shown impressive performance on several new tasks without updating the model's parameters i.e. using zero or a few labeled examples, but the capabilities of LLMs are yet unexplored for feature-specific sentiment analysis. The goal of our study is to explore the capabilities of LLMs to perform feature-specific sentiment analysis of user reviews. This study compares the performance of state-of-the-art LLMs, including GPT-4, ChatGPT, and different variants of Llama-2 chat, against previous approaches for extracting app features and associated sentiments in zero-shot, 1-shot, and 5-shot scenarios. The results indicate that GPT-4 outperforms the rule-based SAFE by 17% in f1-score for extracting app features in the zero-shot scenario, with 5-shot further improving it by 6%. However, the fine-tuned RE-BERT exceeds GPT-4 by 6% in f1-score. For predicting positive and neutral sentiments, GPT-4 achieves f1-scores of 76% and 45% in the zero-shot setting, which improve by 7% and 23% in the 5-shot setting, respectively. Our study conducts a thorough evaluation of both proprietary and open-source LLMs to provide an objective assessment of their performance in extracting feature-sentiment pairs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑