SOUL：迈向语言的情感与观点理解
SOUL: Towards Sentiment and Opinion Understanding of Language
AI总结：
本文提出SOUL任务，通过评论理解与理由生成两个子任务评估语言模型的情感与观点理解能力，并构建新数据集，实验表明现有模型与人类表现差距最高达27%，凸显情感分析仍需进一步突破。
摘要：
情感分析是一项成熟的自然语言处理任务，其中情感极性分类是最流行和最具代表性的任务之一。然而，尽管预训练语言模型在这一领域取得了成功，它们往往难以捕捉情感分析更广泛的复杂性。为了解决这一问题，我们提出了一项名为语言情感与观点理解（SOUL）的新任务。SOUL旨在通过两个子任务来评估情感理解能力：评论理解（RC）和理由生成（JG）。RC要求基于评论文本验证聚焦于主观信息的陈述，而JG则要求模型为其情感预测提供解释。为了实现全面评估，我们标注了一个包含来自3,638条评论的15,028条陈述的新数据集。实验结果表明，SOUL对小型和大型语言模型而言都是一项具有挑战性的任务，与人类表现相比，性能差距最高可达27%。此外，由人类专家和GPT-4共同进行的评估凸显了小型语言模型在生成基于推理的理由方面的局限性。这些发现强调了SOUL任务对现有模型的挑战性，表明需要进一步推进情感分析研究以应对其复杂性。新数据集和代码可在https://github.com/DAMO-NLP-SG/SOUL获取。
英文摘要：
Sentiment analysis is a well-established natural language processing task, with sentiment polarity classification being one of its most popular and representative tasks. However, despite the success of pre-trained language models in this area, they often fall short of capturing the broader complexities of sentiment analysis. To address this issue, we propose a new task called Sentiment and Opinion Understanding of Language (SOUL). SOUL aims to evaluate sentiment understanding through two subtasks: Review Comprehension (RC) and Justification Generation (JG). RC seeks to validate statements that focus on subjective information based on a review text, while JG requires models to provide explanations for their sentiment predictions. To enable comprehensive evaluation, we annotate a new dataset comprising 15,028 statements from 3,638 reviews. Experimental results indicate that SOUL is a challenging task for both small and large language models, with a performance gap of up to 27% when compared to human performance. Furthermore, evaluations conducted with both human experts and GPT-4 highlight the limitations of the small language model in generating reasoning-based justifications. These findings underscore the challenging nature of the SOUL task for existing models, emphasizing the need for further advancements in sentiment analysis to address its complexities. The new dataset and code are available at https://github.com/DAMO-NLP-SG/SOUL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
