FINEST: 通过细粒度评估改进LLM对敏感话题的响应
FINEST: Improving LLM Responses to Sensitive Topics Through Fine-Grained Evaluation
AI总结：
FINEST通过细粒度评估分类法，改进LLM对敏感话题的响应，显著减少错误率并提升帮助性与安全性。
摘要：
大型语言模型（LLMs）在处理敏感话题时往往生成过于谨慎和模糊的响应，牺牲了帮助性以换取安全性。现有的评估框架缺乏系统方法来识别和解决对敏感话题响应中的特定弱点，使得同时改进安全性和帮助性变得困难。为此，我们引入了FINEST，一种针对敏感话题的细粒度响应评估分类法，将帮助性和无害性分解为内容、逻辑和适当性三类错误。在韩国敏感问题数据集上的实验表明，我们的基于评分和错误的改进流程，在FINEST的指导下，显著提高了模型响应在所有三类中的表现，优于无指导的改进。值得注意的是，基于评分的改进——提供类别特定的评分和解释——带来了最大的提升，将适当性错误句子的比例减少了高达33.09%。这项工作为LLM对敏感问题的评估和改进奠定了更可解释和全面的基础。
英文摘要：
Large Language Models (LLMs) often generate overly cautious and vague responses on sensitive topics, sacrificing helpfulness for safety. Existing evaluation frameworks lack systematic methods to identify and address specific weaknesses in responses to sensitive topics, making it difficult to improve both safety and helpfulness simultaneously. To address this, we introduce FINEST, a FINE-grained response evaluation taxonomy for Sensitive Topics, which breaks down helpfulness and harmlessness into errors across three main categories: Content, Logic, and Appropriateness. Experiments on a Korean-sensitive question dataset demonstrate that our score- and error-based improvement pipeline, guided by FINEST, significantly improves the model responses across all three categories, outperforming refinement without guidance. Notably, score-based improvement -- providing category-specific scores and justifications -- yields the most significant gains, reducing the error sentence ratio for Appropriateness by up to 33.09%. This work lays the foundation for a more explainable and comprehensive evaluation and improvement of LLM responses to sensitive questions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
