Despite the success of existing referenced metrics (e.g., BLEU and MoverScore), they correlate poorly with human judgments for open-ended text generation including story or dialog generation because of the notorious one-to-many issue: there are many plausible outputs for the same input, which may differ substantially in literal or semantics from the limited number of given references. To alleviate this issue, we propose UNION, a learnable unreferenced metric for evaluating open-ended story generation, which measures the quality of a generated story without any reference. Built on top of BERT, UNION is trained to distinguish human-written stories from negative samples and recover the perturbation in negative stories. We propose an approach of constructing negative samples by mimicking the errors commonly observed in existing NLG models, including repeated plots, conflicting logic, and long-range incoherence. Experiments on two story datasets demonstrate that UNION is a reliable measure for evaluating the quality of generated stories, which correlates better with human judgments and is more generalizable than existing state-of-the-art metrics.
When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations
The University of Hong Kong（香港大学）；Peking University（北京大学）；Tsinghua University（清华大学）；Beijing Institute of Mathematical Sciences and Applications (BIMSA)（北京数学科学与应用研究院）
Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence
Tsinghua University（清华大学）；Institute for Artificial Intelligence（人工智能研究院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Netease Fuxi AI Lab（网易伏羲人工智能实验室）；TAL Education Group（好未来教育集团）
If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs
University of Electronic Science and Technology of China（电子科技大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能重点实验室）；Nanyang Technological University（南洋理工大学）；Peking University（北京大学）；Spin Matrix, China（中国Spin Matrix公司）
