通过思维链提示探究大型语言模型在反思性评估方法中的效能
Investigating the Efficacy of Large Language Models in Reflective Assessment Methods through Chain of Thoughts Prompting
AI总结：
本研究评估四个 LLM 借助 CoT 提示评分医学生反思性文章及批判性思维的能力，发现 ChatGPT 表现最佳（Cohen kappa 为 0.53），Llama-7b 最差。
摘要：
大型语言模型，例如 Generative Pre-trained Transformer 3（又称 GPT-3），已被开发用于通过分析大量文本数据来理解语言，从而识别词语之间的模式和联系。尽管 LLM 在各种与文本相关的任务中展现出令人瞩目的性能，但它们在与推理相关的任务上仍面临挑战。为应对这一挑战，Chain of Thought（CoT，思维链）提示方法被提出，用以提升 LLM 在复杂推理任务中的能力，例如求解数学应用题以及基于逻辑论证推理回答问题。本研究的主要目的是评估四个语言模型对三年级医学生反思性文章的评分能力。评估将特别聚焦于使用 CoT 提示来评价批判性思维技能。本研究将作出以下贡献：介绍并说明如何指导模型基于一个其此前未曾训练过的数据集来评估反思性文章；阐述如何将 CoT 提示作为一种教学方法，用于训练大型模型执行特定任务。我们的结果表明，在所有模型中，Llama-7b 的表现最差，呈现出最高的均方误差。相反，ChatGPT 成为表现最优的模型，拥有 0.53 的更高 Cohen kappa 分数值。最后，需要指出的是，所选模型确实通过允许用户删除其自己进行过的对话来优先考虑用户隐私。
英文摘要：
Large Language Models, such as Generative Pre-trained Transformer 3 (aka. GPT-3), have been developed to understand language through the analysis of extensive text data, allowing them to identify patterns and connections between words. While LLMs have demonstrated impressive performance across various text-related tasks, they encounter challenges in tasks associated with reasoning. To address this challenge, Chain of Thought(CoT) prompting method has been proposed as a means to enhance LLMs' proficiency in complex reasoning tasks like solving math word problems and answering questions based on logical argumentative reasoning. The primary aim of this research is to assess how well four language models can grade reflective essays of third-year medical students. The assessment will specifically target the evaluation of critical thinking skills using CoT prompting. The research will provide the following contributions; to introduce and educate on the process of instructing models to evaluate reflective essays from a dataset they have not been previously trained on; to illustrate the use of CoT prompting as an instructional approach for training large models to carry out particular tasks. Our results suggest that among all the models, Llama-7b performs the least effectively, displaying the highest mean squared error. Conversely, ChatGPT emerges as the superior model, boasting a higher Cohen kappa score value of 0.53. Lastly, it's important to note that the selected models do prioritise user privacy by allowing users to delete their own conducted conversations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
