NovAScore：一种用于评估文档级新颖性的新型自动化指标
NovAScore: A New Automated Metric for Evaluating Document Level Novelty
AI总结：
针对信息冗余下新颖性检测研究不足、人工标注成本高的问题，提出自动化文档级新颖性评估指标NovAScore，通过聚合原子信息的新颖性与显著性分数并结合动态权重调整，实现与人类判断高度相关的评估结果。
摘要：
在线内容的快速扩张加剧了信息冗余问题，凸显了对能够识别真正新信息的解决方案的需求。尽管存在这一挑战，研究界对新颖性检测的关注却有所下降，尤其是在大语言模型（LLMs）兴起的背景下。此外，以往的方法严重依赖人工标注，这种方式耗时且成本高昂，当标注人员必须将目标文档与大量历史文档进行比较时，难度尤其大。在本研究中，我们提出了NovAScore（原子性新颖性评估评分），这是一种用于评估文档级新颖性的自动化指标。NovAScore聚合原子信息的新颖性和显著性分数，具备高可解释性，能够对文档的新颖性进行细致分析。凭借其动态权重调整机制，NovAScore灵活性更强，为评估文档内信息的新颖程度和重要性提供了额外维度。我们的实验表明，NovAScore与人类对新颖性的判断高度相关，在TAP-DLND 1.0数据集上达到0.626的点二列相关系数，在内部人工标注数据集上达到0.920的皮尔逊相关系数。
英文摘要：
The rapid expansion of online content has intensified the issue of information redundancy, underscoring the need for solutions that can identify genuinely new information. Despite this challenge, the research community has seen a decline in focus on novelty detection, particularly with the rise of large language models (LLMs). Additionally, previous approaches have relied heavily on human annotation, which is time-consuming, costly, and particularly challenging when annotators must compare a target document against a vast number of historical documents. In this work, we introduce NovAScore (Novelty Evaluation in Atomicity Score), an automated metric for evaluating document-level novelty. NovAScore aggregates the novelty and salience scores of atomic information, providing high interpretability and a detailed analysis of a document's novelty. With its dynamic weight adjustment scheme, NovAScore offers enhanced flexibility and an additional dimension to assess both the novelty level and the importance of information within a document. Our experiments show that NovAScore strongly correlates with human judgments of novelty, achieving a 0.626 Point-Biserial correlation on the TAP-DLND 1.0 dataset and a 0.920 Pearson correlation on an internal human-annotated dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
