超越准确率：基于可解释性的有害内容检测分析
Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection
AI总结：
本文基于可解释性分析有害内容检测模型的行为，揭示了聚合评估指标无法发现的局限性，强调可解释性在提升透明度和诊断能力中的作用。
摘要：
尽管自动化有害内容检测系统被广泛用于监控在线平台，但审核员和终端用户往往无法理解其预测背后的逻辑。尽管近期研究侧重于提高分类准确率，但对神经网络模型为何将内容标记为有害，尤其是边界、上下文和政治敏感情况，关注较少。本文分析了一个基于Civil Comments数据集训练的神经有害内容检测模型，使用Shapley Additive Explanations和Integrated Gradients两种后验解释方法，研究RoBERTa分类器在正确预测和系统性失败案例中的行为。尽管整体表现优异，AUC为0.93，准确率为0.94，分析揭示了聚合评估指标无法观察到的局限性。Integrated Gradients似乎提取了更多模糊的上下文属性，而Shapley Additive Explanations则提取了更多聚焦于显性词汇线索的属性。这两种方法输出的分歧表现在假阴性和假阳性中。定性案例研究揭示了重复失败模式，如间接毒性、词汇过度归因或政治讨论。结果表明，可解释AI可以通过暴露模型不确定性并增加自动化决策的可解释性来促进人工审核。最重要的是，本文强调可解释性作为在线有害内容检测系统透明度和诊断资源的作用，而非提升性能的杠杆。
英文摘要：
Although automated harmful content detection systems are frequently used to monitor online platforms, moderators and end users frequently cannot understand the logic underlying their predictions. While recent studies have focused on increasing classification accuracy, little focus has been placed on comprehending why neural models identify content as harmful, especially when it comes to borderline, contextual, and politically sensitive situations. In this work, a neural harmful content detection model trained on the Civil Comments dataset is analyzed explainability-drivenly. Two popular post-hoc explanation methods, Shapley Additive Explanations and Integrated Gradients, are used to analyze the behavior of a RoBERTa-based classifier in both correct predictions and systematic failure cases. Despite strong overall performance, with an area under the curve of 0.93 and an accuracy of 0.94, the analysis reveals limitations that are not observable from aggregate evaluation metrics alone. Integrated Gradients appear to extract more diffuse contextual attributions while Shapley Additive Explanations extract more focused attributions on explicit lexical cues. The consequent divergence in their outputs manifests in both false negatives and false positives. Qualitative case studies reveal recurring failure modes such as indirect toxicity, lexical over-attribution, or political discourse. The results suggest that explainable AI can foster human-in-the-loop moderation by exposing model uncertainty and increasing the interpretable rationale behind automated decisions. Most importantly, this work highlights the role of explainability as a transparency and diagnostic resource for online harmful content detection systems rather than as a performance-enhancing lever.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
