基于提示的清晰度评估与政治问答主题检测
Prompt-Based Clarity Evaluation and Topic Detection in Political Question Answering
AI总结：
本文研究了基于提示的清晰度评估与政治问答主题检测，发现思维链提示显著提升清晰度和主题识别准确性，但细粒度逃避检测仍具挑战性。
摘要：
自动评估大语言模型（LLM）响应需要不仅事实正确性，还需要清晰性，特别是在政治问答中。尽管最近的数据集提供了人类注释的清晰性和逃避性，但提示设计对自动清晰度评估的影响仍被低估。在本文中，我们使用SemEval 2026共享任务中的CLARITY数据集研究基于提示的清晰度评估。我们比较了数据集提供的GPT-3.5基线模型，与在三种提示策略下评估的GPT-5.2模型：简单提示、思维链提示和思维链加少量示例提示。模型预测通过准确率和类别级指标以及层次化精确匹配与人类注释进行评估。结果表明，GPT-5.2在清晰度预测上始终优于GPT-3.5基线，使用思维链加少量示例提示时，准确率从56%提升至63%。思维链提示在逃避准确性上达到34%，但改进在细粒度逃避类别中稳定性较低。我们进一步评估了主题识别，发现基于推理的提示将准确性从60%提升到74%，相对人类注释。总体而言，我们的发现表明，提示设计能够可靠地提高高层清晰度评估，但细粒度逃避和主题检测在结构化推理提示下仍具有挑战性。
英文摘要：
Automatic evaluation of large language model (LLM) responses requires not only factual correctness but also clarity, particularly in political question-answering. While recent datasets provide human annotations for clarity and evasion, the impact of prompt design on automatic clarity evaluation remains underexplored. In this paper, we study prompt-based clarity evaluation using the CLARITY dataset from the SemEval 2026 shared task. We compare a GPT-3.5 baseline provided with the dataset against GPT-5.2 evaluated under three prompting strategies: simple prompting, chain-of-thought prompting, and chain-of-thought with few-shot examples. Model predictions are evaluated against human annotations using accuracy and class-wise metrics for clarity and evasion, along with hierarchical exact match. Results show that GPT-5.2 consistently outperforms the GPT-3.5 baseline on clarity prediction, with accuracy improving from 56 percent to 63 percent under chain-of-thought with few-shot prompting. Chain-of-thought prompting yields the highest evasion accuracy at 34 percent, though improvements are less stable across fine-grained evasion categories. We further evaluate topic identification and find that reasoning-based prompting improves accuracy from 60 percent to 74 percent relative to human annotations. Overall, our findings indicate that prompt design reliably improves high-level clarity evaluation, while fine-grained evasion and topic detection remain challenging despite structured reasoning prompts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
