arXiv 2311.072372024-10-04cs.CLcs.AI

寻找长尾：通过逻辑规则引导搜索系统化生成长尾推理知识

In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search

University of Southern California（南加州大学）；Ohio State University（俄亥俄州立大学）；Fudan University（复旦大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Pittsburgh（匹兹堡大学）；University of Washington（华盛顿大学）；Cornell University（康奈尔大学）

Huihan Li, Yuting Ning, Zeyi Liao, Siyuan Wang, Xiang Lorraine Li, Ximing Lu, Wenting Zhao, Faeze Brahman, Yejin Choi, Xiang Ren

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LINK框架，通过逻辑规则引导搜索系统化生成长尾推理数据，并构建含108K条语句的LINT数据集，评估发现现有LLM在长尾分布上性能显著下降，凸显长尾评估对提升模型泛化能力的重要性。

摘要：

为了有效地将大语言模型（LLM）用于真实世界查询，它们必须能够泛化到长尾分布，即模型表现出低置信度的稀有示例。在这项工作中，我们迈出了在推理知识的长尾分布中评估LLM的第一步。我们以自然语言推理任务为例展示长尾评估。首先，我们引入了Logic-Induced-Knowledge-Search（LINK），一个系统化的长尾数据生成框架，用于获得事实正确但属于长尾的推理语句。LINK使用基于符号规则的逐变量提示来寻找低置信度语句，同时确保事实正确性。然后，我们使用LINK构建了Logic-Induced-Long-Tail（LINT），一个大规模长尾推理知识数据集，包含跨越四个领域的108K条语句。我们在LINT上评估了流行的LLM；我们发现，与头部分布数据相比，最先进的LLM在长尾数据上表现出显著的性能下降（GPT4相对下降21%），而较小的模型表现出更严重的泛化弱点。这些结果进一步强调了长尾评估在开发可泛化LLM中的必要性。

英文摘要：

To effectively use large language models (LLMs) for real-world queries, it is imperative that they generalize to the long-tail distribution, i.e. rare examples where models exhibit low confidence. In this work, we take the first step towards evaluating LLMs in the long-tail distribution of inferential knowledge. We exemplify long-tail evaluation on the Natural Language Inference task. First, we introduce Logic-Induced-Knowledge-Search (LINK), a systematic long-tail data generation framework, to obtain factually-correct yet long-tail inferential statements. LINK uses variable-wise prompting grounded on symbolic rules to seek low-confidence statements while ensuring factual correctness. We then use LINK to curate Logic-Induced-Long-Tail (LINT), a large-scale long-tail inferential knowledge dataset that contains 108K statements spanning four domains. We evaluate popular LLMs on LINT; we find that state-of-the-art LLMs show significant performance drop (21% relative drop for GPT4) on long-tail data as compared to on head distribution data, and smaller models show even more generalization weakness. These results further underscore the necessity of long-tail evaluation in developing generalizable LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。