Reasoning Factual Knowledge in Structured Data with Large Language Models
Hong Kong Polytechnic University(香港理工大学);University Technology Sydney(悉尼科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Soochow University(苏州大学);The Education University of Hong Kong(香港教育大学)
Large language models (LLMs) have made remarkable progress in various natural language processing tasks as a benefit of their capability to comprehend and reason with factual knowledge. However, a significant amount of factual knowledge is stored in structured data, which possesses unique characteristics that differ from the unstructured texts used for pretraining. This difference can introduce imperceptible inference parameter deviations, posing challenges for LLMs in effectively utilizing and reasoning with structured data to accurately infer factual knowledge. To this end, we propose a benchmark named StructFact, to evaluate the structural reasoning capabilities of LLMs in inferring factual knowledge. StructFact comprises 8,340 factual questions encompassing various tasks, domains, timelines, and regions. This benchmark allows us to investigate the capability of LLMs across five factual tasks derived from the unique characteristics of structural facts. Extensive experiments on a set of LLMs with different training strategies reveal the limitations of current LLMs in inferring factual knowledge from structured data. We present this benchmark as a compass to navigate the strengths and weaknesses of LLMs in reasoning with structured data for knowledge-sensitive tasks, and to encourage advancements in related real-world applications. Please find our code at https://github.com/EganGu/StructFact.
Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Wenge Technology Co., Ltd(北京文格科技有限公司)
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Post and Telecommunications(北京邮电大学)
FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning
Massachusetts Institute of Technology(麻省理工学院);Havard Univeristy(哈佛大学);LG CNS;Université de Montréal(蒙特利尔大学);Mila(米拉研究所);Google(谷歌);Stanford University(斯坦福大学)
Co-occurrence is not Factual Association in Language Models
Department of Electronics Engineering(电子工程系);Tsinghua University(清华大学);Department of Electronics Engineering, College of AI(人工智能学院电子工程系);Beijing National Research Center for Information Science and Technology(北京信息科学国家研究中心);Center for Big Data and Clinical Research, Institute for Precision Medicine(精准医学研究院大数据与临床研究中心)
MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
UC Santa Cruz(加州大学圣克鲁兹分校);University of British Columbia(不列颠哥伦比亚大学);Nanyang Technological University(南洋理工大学);Stanford University(斯坦福大学);Weill Cornell Medicine(威尔康奈尔医学院);NYU Langone Health(纽约大学兰贡医学中心);Chungnam National University Sejong Hospital(忠南大学世宗医院);Vector Institute(矢量研究院);Pusan National University Hospital(釜山国立大学医院);Massachusetts General Hospital(麻省总医院)
MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
Data Clinic, University Hospital of Nantes, France(南特大学医院数据诊所,法国);Nantes Université, École Centrale Nantes, CNRS, LS2N, France(南特大学,中央理工学院南特分校,国家科学研究中心,LS2N,法国)
Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs
Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全重点实验室);Shandong University(山东大学);Shandong University-Weihai Research Institute of Industrial Technology(山东大学威海工业技术研究院)
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);State Key Laboratory of Computer Science(计算机科学国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院)
When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);AI Lab at Lenovo Research(联想研究院人工智能实验室);School of Artificial Intelligence and Data Science, University of International Business and Economics(国际商务与经济大学人工智能与数据科学学院)
RATT: A Thought Structure for Coherent and Correct LLM Reasoning
Portland State University(波特兰州立大学);Renmin University of China(中国人民大学);Pennsylvania State University(宾夕法尼亚州立大学);Dalian Maritime University(大连海事大学);University of Kansas(堪萨斯大学)
Temporal Knowledge Question Answering via Abstract Reasoning Induction
Laboratory for Big Data and Decision, National University of Defense Technology(国防科技大学大数据与决策实验室);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))