好奇心驱动的LLM-as-a-judge用于个性化创意判断
Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment
AI总结：
提出好奇心驱动的LLM-as-a-judge方法，个性化评估创意写作，在TTCW基准上优于监督微调，尤其适用于标注者意见不一致的主观评估。
摘要：
现代大型语言模型（LLMs）在评估数学推理和事实准确性等客观任务上表现出色，但在面对评估创造力的细微、主观性质时却表现不佳。在这项工作中，我们提出了一种新颖的好奇心驱动的LLM-as-a-judge方法，用于评估创意写作，该方法针对每个个体的创意判断进行个性化。我们使用了Chakrabarty等人（2024）引入的托兰斯创造性思维测试（TTCW）基准，该基准包含由人类专家在诸如原创性等多个主观维度上标注的故事，以验证我们的假设。我们证明，我们的方法使不同规模的模型能够学习不同个体的细微创意判断，在皮尔逊相关系数、Cohen's kappa和F1值等多种评估指标上，相较于基线监督微调（SFT）方法显示出改进。我们的方法在并非所有标注者都达成一致的主观评估中尤其有用。
英文摘要：
Modern large language models (LLMs) excel at objective tasks such as evaluating mathematical reasoning and factual accuracy, yet they falter when faced with the nuanced, subjective nature of assessing creativity. In this work, we propose a novel curiosity-driven LLM-as-a-judge for evaluating creative writing which is personlized to each individual's creative judgments. We use the Torrance Test of Creative Thinking(TTCW) benchmark introduced in Chakrabarty et al. (2024), which has stories annotated by expert humans across various subjective dimensions like Originality, to test our hypothesis. We show that our method enables models across various sizes, to learn the nuanced creative judgments of different individuals, by showing improvements over baseline supervised finetuning(SFT) method across various evaluation metrics like Pearson correlation, Cohen's and F1 values. Our method is especially useful in subjective evaluations where not all the annotators agree with each other.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
