基于LLM的论证分类综合研究：从LLAMA到GPT-4o再到Deepseek-R1
A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1
AI总结：
本文在Args.me、UKP等论证数据集上评测GPT、Llama、DeepSeek及Chain-of-Thoughts推理增强模型，发现GPT-4o整体最佳、Deepseek-R1在推理模型中领先，并分析错误与提示算法及数据集缺陷。
摘要：
论证挖掘（AM）是一个跨学科研究领域，融合了逻辑学、哲学、语言学、修辞学、法学、心理学和计算机科学的洞见。它涉及自动识别和提取论证成分（例如前提和主张），以及检测这些成分之间的关系，例如支持、攻击或中立。近年来，该领域取得了显著进展，尤其是随着大型语言模型（LLM）的出现；与传统方法和其他深度学习模型相比，LLM提升了分析和提取论证语义的效率。目前已有许多用于测试和验证LLM质量的基准，但围绕这些模型在公开可用论证分类数据库上的运行情况，仍然缺乏研究和结果。本文对若干LLM进行了研究，使用了Args.me和UKP等多样化数据集。受测模型包括GPT、Llama和DeepSeek的多个版本，以及融入Chain-of-Thoughts算法、具备推理增强能力的变体。结果表明，ChatGPT-4o在论证分类基准上优于其他模型。在融入推理能力的模型中，Deepseek-R1表现出优势。然而，尽管GPT-4o和Deepseek-R1具有优势，它们仍然会犯错。本文讨论了所有模型最常见的错误。据我们所知，所呈现的工作是首次利用LLM和提示算法对上述数据集进行的更广泛分析。该工作还展示了已知提示算法在论证分析中的若干弱点，同时指出了改进方向。本文的附加价值在于对可用论证数据集进行深入分析，并展示其不足。
英文摘要：
Argument mining (AM) is an interdisciplinary research field that integrates insights from logic, philosophy, linguistics, rhetoric, law, psychology, and computer science. It involves the automatic identification and extraction of argumentative components, such as premises and claims, and the detection of relationships between them, such as support, attack, or neutrality. Recently, the field has advanced significantly, especially with the advent of large language models (LLMs), which have enhanced the efficiency of analyzing and extracting argument semantics compared to traditional methods and other deep learning models. There are many benchmarks for testing and verifying the quality of LLM, but there is still a lack of research and results on the operation of these models in publicly available argument classification databases. This paper presents a study of a selection of LLM's, using diverse datasets such as Args.me and UKP. The models tested include versions of GPT, Llama, and DeepSeek, along with reasoning-enhanced variants incorporating the Chain-of-Thoughts algorithm. The results indicate that ChatGPT-4o outperforms the others in the argument classification benchmarks. In case of models incorporated with reasoning capabilities, the Deepseek-R1 shows its superiority. However, despite their superiority, GPT-4o and Deepseek-R1 still make errors. The most common errors are discussed for all models. To our knowledge, the presented work is the first broader analysis of the mentioned datasets using LLM and prompt algorithms. The work also shows some weaknesses of known prompt algorithms in argument analysis, while indicating directions for their improvement. The added value of the work is the in-depth analysis of the available argument datasets and the demonstration of their shortcomings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
