When completing knowledge-intensive tasks, humans sometimes need an answer and a corresponding reference passage for auxiliary reading. Previous methods required obtaining pre-segmented article chunks through additional retrieval models. This paper explores leveraging the parameterized knowledge stored during the pre-training phase of large language models (LLMs) to recall reference passage from any starting position independently. We propose a two-stage framework that simulates the scenario of humans recalling easily forgotten references. Initially, the LLM is prompted to recall document title identifiers to obtain a coarse-grained document set. Then, based on the acquired coarse-grained document set, it recalls fine-grained passage. In the two-stage recall process, we use constrained decoding to ensure that content outside of the stored documents is not generated. To increase speed, we only recall a short prefix in the second stage, and then locate its position to retrieve a complete passage. Experiments on KILT knowledge-sensitive tasks have verified that LLMs can independently recall reference passage locations in various task forms, and the obtained reference significantly assists downstream tasks.
University of Mannheim（曼海姆大学）；University of Technology Nuremberg（纽伦堡技术大学）；National Institute of Information and Communications Technology（信息与通信技术国家研究所）
Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation
Shanghai Institute of Artificial Intelligence for Education（上海人工智能教育研究院）；East China Normal University（华东师范大学）；School of Data Science, Fudan University（复旦大学数据科学学院）；College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Alibaba（阿里巴巴集团）
Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models
Singapore Management University（新加坡管理大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；University of Science and Technology of China（中国科学技术大学）；University of Georgia（佐治亚大学）；Skywork AI（天工智芯（Skywork AI））
DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）；Tongyi Lab, Alibaba Group（阿里巴巴集团通义实验室）
本文提出新任务 Dialogue Element Modeling 和新基准 DEMO，用于全面建模与评估 LLMs 对话系统，并构建了基于模仿学习的 DEMO agent，在对话元素建模及域外任务中表现优异。
E2Edev: Benchmarking Large Language Models in End-to-End Software Development Task
College of Computer Science, Sichuan University（四川大学计算机学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；CHAT NLP Group, Singapore Management University（新加坡国立管理大学CHAT NLP组）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China（教育部长机器学习与产业智能工程研究中心）
FollowTable: A Benchmark for Instruction-Following Table Retrieval
Southeast University（东南大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（新一代人工智能技术及其交叉应用国家重点实验室（东南大学））；Its Interdisciplinary Applications (Southeast University), Ministry of Education（交叉应用（东南大学），教育部）
