CoKe:通过关键词链合理化实现可定制的细粒度故事评估
CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization
AI总结:
针对创意文本评估中自洽性推理的目标错配问题,提出先生成关键词再生成理据的CoKe方法,在StoryER数据集上以小模型实现超GPT-4的人类级评估性能且参数量更低。
摘要:
使用语言模型评估人类撰写的故事等创意文本一直是一项具有挑战性的任务——原因在于多标注者评分存在主观性。为模拟人类的思考过程,思维链(CoT)会生成自由文本解释,用于引导模型的预测,而自洽性(Self-Consistency, SC)则对多条生成解释的预测结果进行边际化处理。本研究发现,广泛使用的自洽性推理方法会产生次优结果,原因在于生成“看似流畅”的解释与实际得出故事某一维度的优质评分预测之间存在目标不匹配问题。为解决这一挑战,我们提出关键词链(Chain-of-Keywords, CoKe)方法,该方法在生成自由文本理据之前先生成关键词序列,以此引导我们的评估语言模型进行评分预测。随后,我们生成多样化的此类关键词集合,并聚合这些生成结果对应的分数。在StoryER数据集上,基于我们的小型微调评估模型的CoKe不仅达到了人类水平的性能,且与人类标注者的相关性较GPT-4提升2倍,显著优于GPT-4,同时所需参数量大幅减少。
英文摘要:
Evaluating creative text such as human-written stories using language models has always been a challenging task -- owing to the subjectivity of multi-annotator ratings. To mimic the thinking process of humans, chain of thought (CoT) generates free-text explanations that help guide a model's predictions and Self-Consistency (SC) marginalizes predictions over multiple generated explanations. In this study, we discover that the widely-used self-consistency reasoning methods cause suboptimal results due to an objective mismatch between generating 'fluent-looking' explanations vs. actually leading to a good rating prediction for an aspect of a story. To overcome this challenge, we propose $\textbf{C}$hain-$\textbf{o}$f-$\textbf{Ke}$ywords (CoKe), that generates a sequence of keywords $\textit{before}$ generating a free-text rationale, that guide the rating prediction of our evaluation language model. Then, we generate a diverse set of such keywords, and aggregate the scores corresponding to these generations. On the StoryER dataset, CoKe based on our small fine-tuned evaluation models not only reach human-level performance and significantly outperform GPT-4 with a 2x boost in correlation with human annotators, but also requires drastically less number of parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。