arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11034cs.CL
收藏

PAT-Questions:一个用于当前锚定时间问答的自更新基准

PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering

University of California, Riverside(加州大学河滨分校)

Jannat Ara Meem, Muhammad Shihab Rashid, Yue Dong, Vagelis Hristidis

AI总结:

本文提出了当前锚定时间问答(PATQA)的自更新基准PAT-Questions,支持通过SPARQL查询自动刷新答案,评估了多个LLM和SOTA模型,揭示了现有方法在处理相对当前时间问题上的局限性。

摘要:

现有的时间问答(TQA)研究主要集中在锚定于特定时间戳或事件的问题上(例如“1970年的美国总统是谁?”)。很少有研究关注时间上下文相对于当前时间的问题(例如“上一任美国总统是谁?”)。我们将此问题称为当前锚定时间问答(PATQA)。PATQA带来了独特的挑战:(1)大型语言模型(LLMs)可能拥有过时的知识,(2)复杂的时间关系(例如“之前”、“上一任”)难以推理,(3)可能需要多跳推理,(4)基准的黄金答案必须持续更新。为了应对这些挑战,我们引入了PAT-Questions基准,其中包含单跳和多跳时间问题。如果可用,PAT-Questions中的答案可以通过在知识图上重新运行SPARQL查询来自动刷新。我们通过直接提示和检索增强生成(RAG)在PAT-Questions上评估了几个最先进的LLM和一个SOTA时间推理模型(TEMPREASON-T5)。结果突出了现有解决方案在PATQA中的局限性,并促使需要新方法来提高PATQA推理能力。

英文摘要:

Existing work on Temporal Question Answering (TQA) has predominantly focused on questions anchored to specific timestamps or events (e.g. "Who was the US president in 1970?"). Little work has studied questions whose temporal context is relative to the present time (e.g. "Who was the previous US president?"). We refer to this problem as Present-Anchored Temporal QA (PATQA). PATQA poses unique challenges: (1) large language models (LLMs) may have outdated knowledge, (2) complex temporal relationships (e.g. 'before', 'previous') are hard to reason, (3) multi-hop reasoning may be required, and (4) the gold answers of benchmarks must be continuously updated. To address these challenges, we introduce the PAT-Questions benchmark, which includes single and multi-hop temporal questions. The answers in PAT-Questions can be automatically refreshed by re-running SPARQL queries on a knowledge graph, if available. We evaluate several state-of-the-art LLMs and a SOTA temporal reasoning model (TEMPREASON-T5) on PAT-Questions through direct prompting and retrieval-augmented generation (RAG). The results highlight the limitations of existing solutions in PATQA and motivate the need for new methods to improve PATQA reasoning capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑