LLM-as-a-Fuzzy-Judge：基于模糊逻辑微调大语言模型作为临床评估裁判
LLM-as-a-Fuzzy-Judge: Fine-Tuning Large Language Models as a Clinical Evaluation Judge with Fuzzy Logic
AI总结：
针对医学教育中临床技能自动化评估难与医生主观偏好对齐的问题，提出结合模糊逻辑微调LLM的LLM-as-a-Fuzzy-Judge方法，实现超80%的整体准确率及超90%的主要项准确率，推动医学教育自动化评估发展。
摘要：
临床沟通技能在医学教育中至关重要，而大规模开展临床沟通技能的练习与评估颇具挑战。尽管基于大语言模型（LLM）的临床情景模拟在提升医学生临床实践方面已展现出潜力，但要实现符合医生细致判断标准的自动化、可扩展临床评估仍十分困难。本文结合模糊逻辑与大语言模型（LLM），提出LLM-as-a-Fuzzy-Judge方法，以解决医学生临床技能自动化评估与医生主观偏好对齐的难题。LLM-as-a-Fuzzy-Judge是一种对LLM进行微调的方法，使其基于四个模糊集（专业性、医学相关性、伦理行为、情境干扰）的人工标注，评估学生与AI患者对话脚本中医学生的表述。本文的研究方法流程为：从基于LLM的医学教育系统中收集数据，基于多维模糊集进行数据标注，随后开展提示工程，并利用这些人工标注对预训练LLM进行监督微调（SFT）。结果显示，LLM-as-a-Fuzzy-Judge的准确率超过80%，主要评估项准确率超过90%，有效结合模糊逻辑与LLM，提供了可解释、与人类对齐的评估方案。本研究表明了结合模糊逻辑与LLM实现人类偏好对齐的可行性，推动了医学教育中的自动化评估发展，为更可靠的评估与判断实践提供支持。本研究的GitHub仓库位于https://github.com/2sigmaEdTech/LLMAsAJudge
英文摘要：
Clinical communication skills are critical in medical education, and practicing and assessing clinical communication skills on a scale is challenging. Although LLM-powered clinical scenario simulations have shown promise in enhancing medical students' clinical practice, providing automated and scalable clinical evaluation that follows nuanced physician judgment is difficult. This paper combines fuzzy logic and Large Language Model (LLM) and proposes LLM-as-a-Fuzzy-Judge to address the challenge of aligning the automated evaluation of medical students' clinical skills with subjective physicians' preferences. LLM-as-a-Fuzzy-Judge is an approach that LLM is fine-tuned to evaluate medical students' utterances within student-AI patient conversation scripts based on human annotations from four fuzzy sets, including Professionalism, Medical Relevance, Ethical Behavior, and Contextual Distraction. The methodology of this paper started from data collection from the LLM-powered medical education system, data annotation based on multidimensional fuzzy sets, followed by prompt engineering and the supervised fine-tuning (SFT) of the pre-trained LLMs using these human annotations. The results show that the LLM-as-a-Fuzzy-Judge achieves over 80\% accuracy, with major criteria items over 90\%, effectively leveraging fuzzy logic and LLM as a solution to deliver interpretable, human-aligned assessment. This work suggests the viability of leveraging fuzzy logic and LLM to align with human preferences, advances automated evaluation in medical education, and supports more robust assessment and judgment practices. The GitHub repository of this work is available at https://github.com/2sigmaEdTech/LLMAsAJudge
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
