用分析方法评估教育LLM：关于反馈中性别偏见的案例研究
Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback
AI总结：
本文通过600篇真实学生作文，利用嵌入式基准框架检测LLM在形成性反馈中的偏见，发现隐性性别提示比显性提示更易引发语义偏移，揭示了LLM在反馈中持续存在的性别偏见。
摘要：
随着教师越来越多地在教育实践中使用生成式人工智能，我们需要稳健的方法来为教学目的基准大型语言模型（LLMs）。本文提出了一种基于嵌入的基准框架，用于检测LLM在形成性反馈中的偏见。使用来自AES 2.0语料库的600篇真实学生作文，我们沿着两个维度构建了受控的反事实：（i）通过基于词典的性别词交换来获取隐性提示，（ii）通过提示中的性别作者背景获取显性提示。我们研究了六个代表性的LLM（即GPT-5 mini、GPT-4o mini、DeepSeek-R1、DeepSeek-R1-Qwen、Gemini 2.5 Pro、Llama-3-8B）。我们首先通过句子嵌入的余弦和欧几里得距离量化响应差异，然后通过排列检验评估显著性，最后通过降维可视化结构。在所有模型中，隐性操作可靠地引发了男性-女性反事实比女性-男性更大的语义偏移。只有GPT和Llama模型对显性性别提示表现出敏感性。这些发现表明，即使是最先进的LLM也表现出对性别替换的非对称语义响应，表明其提供的反馈中存在持续的性别偏见。定性分析进一步揭示了持续的语言差异（例如，男性提示下更支持自主性反馈，而女性提示下更倾向于控制反馈）。我们讨论了公平审计教育生成式人工智能的含义，提出了学习分析中反事实评估的报告标准，并概述了提示设计和部署的实际指导，以确保公平的反馈。
英文摘要：
As teachers increasingly turn to GenAI in their educational practice, we need robust methods to benchmark large language models (LLMs) for pedagogical purposes. This article presents an embedding-based benchmarking framework to detect bias in LLMs in the context of formative feedback. Using 600 authentic student essays from the AES 2.0 corpus, we constructed controlled counterfactuals along two dimensions: (i) implicit cues via lexicon-based swaps of gendered terms within essays, and (ii) explicit cues via gendered author background in the prompt. We investigated six representative LLMs (i.e. GPT-5 mini, GPT-4o mini, DeepSeek-R1, DeepSeek-R1-Qwen, Gemini 2.5 Pro, Llama-3-8B). We first quantified the response divergence with cosine and Euclidean distances over sentence embeddings, then assessed significance via permutation tests, and finally, visualised structure using dimensionality reduction. In all models, implicit manipulations reliably induced larger semantic shifts for male-female counterfactuals than for female-male. Only the GPT and Llama models showed sensitivity to explicit gender cues. These findings show that even state-of-the-art LLMs exhibit asymmetric semantic responses to gender substitutions, suggesting persistent gender biases in feedback they provide learners. Qualitative analyses further revealed consistent linguistic differences (e.g., more autonomy-supportive feedback under male cues vs. more controlling feedback under female cues). We discuss implications for fairness auditing of pedagogical GenAI, propose reporting standards for counterfactual evaluation in learning analytics, and outline practical guidance for prompt design and deployment to safeguard equitable feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
