COGNAC在SemEval-2026任务5中的应用：用于挑战性叙事中人类水平词义可信度评估的LLM集成
COGNAC at SemEval-2026 Task 5: LLM Ensembles for Human-Level Word Sense Plausibility Rating in Challenging Narratives
AI总结：
本文提出使用多个闭源商业LLM的集成方法，通过三种提示策略提升短篇故事中多义词词义可信度评估的准确性，最终在比赛中获得第四名。
摘要：
我们描述了SemEval-2026任务5的系统，该任务要求对短篇故事中给定的多义词词义可信度进行5分Likert量表评分。系统通过准确率（在人类判断均值一个标准差内）和Spearman秩相关系数的加权平均进行评估。我们探索了三种提示策略：（i）基线零样本设置，（ii）带有结构化推理的Chain-of-Thought（CoT）提示，以及（iii）用于同时评估候选词义的比较提示策略。此外，为应对金标签中显著的标注者间差异，我们提出通过平均模型预测进行集成。我们的最佳官方系统，由三种提示策略下的LLM集成组成，在比赛中获得第四名，准确率为0.88，Spearman rho为0.83（平均0.86）。赛后使用额外模型的实验进一步将性能提升至准确率0.92和Spearman rho 0.85（平均0.89）。我们发现比较提示策略在所有模型家族中均提升了性能，而模型集成显著增强了与人类判断均值的一致性，表明LLM集成尤其适合涉及多标注者的主观语义评估任务。
英文摘要：
We describe our system for SemEval-2026 Task 5, which requires rating the plausibility of given word senses of homonyms in short stories on a 5-point Likert scale. Systems are evaluated by the unweighted average of accuracy (within one standard deviation of mean human judgments) and Spearman Rank Correlation. We explore three prompting strategies using multiple closed-source commercial LLMs: (i) a baseline zero-shot setup, (ii) Chain-of-Thought (CoT) style prompting with structured reasoning, and (iii) a comparative prompting strategy for evaluating candidate word senses simultaneously. Furthermore, to account for the substantial inter-annotator variation present in the gold labels, we propose an ensemble setup by averaging model predictions. Our best official system, comprising an ensemble of LLMs across all three prompting strategies, placed 4th on the competition leaderboard with 0.88 accuracy and 0.83 Spearman's rho (0.86 average). Post-competition experiments with additional models further improved this performance to 0.92 accuracy and 0.85 Spearman's rho (0.89 average). We find that comparative prompting consistently improved performance across model families, and model ensembling significantly enhanced alignment with mean human judgments, suggesting that LLM ensembles are especially well suited for subjective semantic evaluation tasks involving multiple annotators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
