面向大型语言模型的时间一致性事实探测
Temporally Consistent Factuality Probing for Large Language Models
AI总结：
本文提出TeCFaP时间一致性事实探测任务与TEMP-COFAC数据集，扩展评估指标，并通过结合MT-IT和CTSRL的CoTSeLF框架提升LLM跨时间查询改写的事实一致性。
摘要：
大型语言模型（LLM）作为替代知识库被大量使用，这要求它们具备事实一致性，即对于改写查询必须同时具有正确性和一致性特征。近来，已有大量工作尝试通过基准数据集和指标评估LLM的这些特征。然而，其查询构造在结构上过于简单（主语-关系-宾语），并存在当代关联，限制了事实性与一致性的更广泛定义。在本研究中，我们提出TeCFaP，这是一种新颖的时间一致性事实探测任务，旨在将一致性事实探测扩展到时间维度。为此，我们构建了TEMP-COFAC，一个由前缀式英语查询改写组成的高质量数据集。随后，我们扩展了现有指标的定义，以表征跨时间维度的一致性事实性。我们对多种LLM进行实验，发现其中大多数在TeCFaP上表现不佳。接下来，我们提出一种新颖解决方案CoTSeLF（Consistent-Time-Sensitive Learning Framework，一致性时间敏感学习框架），将多任务指令微调（MT-IT）与一致性时间敏感强化学习（CTSRL）相结合，以提升LLM的时间一致性事实性。实验表明，CoTSeLF相比若干基线方法具有有效性。
英文摘要：
The prolific use of Large Language Models (LLMs) as an alternate knowledge base requires them to be factually consistent, necessitating both correctness and consistency traits for paraphrased queries. Recently, significant attempts have been made to benchmark datasets and metrics to evaluate LLMs for these traits. However, structural simplicity (subject-relation-object) and contemporary association in their query formulation limit the broader definition of factuality and consistency. In this study, we introduce TeCFaP, a novel Temporally Consistent Factuality Probe task to expand the consistent factuality probe in the temporal dimension. To this end, we propose TEMP-COFAC, a high-quality dataset of prefix-style English query paraphrases. Subsequently, we extend the definitions of existing metrics to represent consistent factuality across temporal dimension. We experiment with a diverse set of LLMs and find most of them performing poorly on TeCFaP. Next, we propose a novel solution CoTSeLF (Consistent-Time-Sensitive Learning Framework) combining multi-task instruction tuning (MT-IT) with consistent-time-sensitive reinforcement learning (CTSRL) to improve temporally consistent factuality in LLMs. Our experiments demonstrate the efficacy of CoTSeLF over several baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
