Large Language Models (LLMs) are increasingly ubiquitous, yet their ability to retain and reason about temporal information remains limited, hindering their application in real-world scenarios where understanding the sequential nature of events is crucial. Our study experiments with 12 state-of-the-art models (ranging from 2B to 70B+ parameters) on a novel numerical-temporal dataset, \textbf{TempUN}, spanning from 10,000 BCE to 2100 CE, to uncover significant temporal retention and comprehension limitations. We propose six metrics to assess three learning paradigms to enhance temporal knowledge acquisition. Our findings reveal that open-source models exhibit knowledge gaps more frequently, suggesting a trade-off between limited knowledge and incorrect responses. Additionally, various fine-tuning approaches significantly improved performance, reducing incorrect outputs and impacting the identification of 'information not available' in the generations. The associated dataset and code are available at (https://github.com/lingoiitgn/TempUN).
Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(智能信息处理重点实验室,计算技术研究所,中国科学院);Key Laboratory of AI Safety, Chinese Academy of Sciences(人工智能安全重点实验室,中国科学院);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Li Auto Inc.(利亚德公司)
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
School of Intelligence Science and Technology, Peking University(北京理工大学智能科学与技术学院);Wuhan University(武汉大学);Shanghai Jiao Tong University(上海交通大学);Nanyang Technological University(南洋理工大学);CASIA(中国科学院自动化研究所);University of Tokyo(东京大学);University of Liverpool(利物浦大学);Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);UC Merced(加州大学默塞德分校)
Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory
TMLR Group, Hong Kong Baptist University(香港 Baptist 大学 TMLR 实验组);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Shanghai Jiao Tong University(上海交通大学);Sydney AI Center, The University of Sydney(悉尼大学悉尼人工智能中心)
MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning
Alibaba Group(阿里巴巴集团);University of Science and Technology of China(中国科学技术大学);Institute of Dataspace, Hefei Comprehensive National Science Center(合肥综合性国家科学中心天枢研究院)
CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process
Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);Munich Research Center, Huawei Technologies(华为技术有限公司慕尼黑研究中心);School of Computer Science, Wuhan University(武汉大学计算机学院);Munich Center for Machine Learning(慕尼黑机器学习中心)
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院)
CLadder: Assessing Causal Reasoning in Language Models
MPI for Intelligent Systems, Tübingen(马克斯·普朗克智能系统研究所,蒂宾根);ETH Zürich(苏黎世联邦理工学院);IIT Kharagpur(印度理工学院卡拉格普尔分校);University of Hong Kong(香港大学);University of Washington(华盛顿大学)
Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks
University of Oxford(牛津大学);Microsoft Research(微软研究院);The Alan Turing Institute(艾伦·图灵研究所);Allen Institute for AI(人工智能研究所);University of Washington(华盛顿大学);Microsoft Corporation(微软公司);University of York(约克大学)
Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution
AGI Institute, School of Computer Science, Shanghai Jiao Tong University(AGI研究院,计算机科学学院,上海交通大学);Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团);Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University(上海市教育委员会智能交互与认知工程重点实验室,上海交通大学)
Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification
University of Warwick(沃里克大学);Queen Mary University of London(伦敦大学女王学院);University of Bologna(博洛尼亚大学);King’s College London(伦敦大学国王学院);The Alan Turing Institute(艾伦·图灵研究所)
LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues
Faculty of Information Technology, Monash University, Australia(墨尔本大学信息技术学院,澳大利亚);School of Information Technology, Monash University Malaysia(墨尔本大学马来西亚分校信息学院);School of Business, Monash University Malaysia(墨尔本大学马来西亚分校商学院);Faculty of Law, Monash University, Australia(墨尔本大学法学院,澳大利亚)
Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning
Strong AI Lab, NAOInstitute, Waipapa Taumata Rau - The University of Auckland(强人工智能实验室、NAO研究所、怀帕帕·陶马陶·劳 - 奥克兰大学);Xtracta, New Zealand(Xtracta公司);School of Computer Science and Engineering, Sun Yat-Sen University(中山大学计算机科学与工程学院)
REMEMBER: Retrieval-based Explainable Multimodal Evidence-guided Modeling for Brain Evaluation and Reasoning in Zero- and Few-shot Neurodegenerative Diagnosis
Lausanne University Hospital (CHUV)(洛桑大学医院(CHUV));University of Lausanne (UNIL)(洛桑大学(UNIL));Vietnam National University(越南国立大学);University of Science, Vietnam National University Ho Chi Minh City(越南国立大学胡志明市科学大学);International University, Vietnam National University Ho Chi Minh City(越南国立大学胡志明市国际大学)
Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);The University of Hong Kong(香港大学)
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
Department of Informatics(信息学院);King’s College London(伦敦国王学院);Institute of Psychology(心理学研究所);University of Tartu(塔尔图大学);Department of Psychology(心理学系);Stanford University(斯坦福大学);The Alan Turing Institute(艾伦·图灵研究所)
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
GeoDS Lab, Department of Geography, University of Wisconsin-Madison, USA(威斯康星大学麦迪逊分校地理系GeoDS实验室);Department of Geography and Regional Research, University of Vienna, Austria(维也纳大学地理与区域研究系);Department of Geography, University of California-Santa Barbara, USA(加州大学圣塔芭芭拉分校地理系)
Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors
Meta;Mila-Quebec AI Institute, University of Montreal(魁北克AI研究所,蒙特利尔大学);Princeton University(普林斯顿大学);Mila--Quebec AI Institute, Université de Montréal(魁北克AI研究所,蒙特利尔大学)
Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering
LTCI, Télécom-Paris, Institut Polytechnique de Paris(LTCI, Télécom-Paris, Institut Polytechnique de Paris);Inria, LJK, Univ. Grenoble Alpes(Inria, LJK, 火车头 Grenoble Alpes);Universitat Autónoma de Barcelona(Autonomous University of Barcelona)
Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve
Technical University of Munich(慕尼黑技术大学);TUM University Hospital(TUM大学医院);LMU Munich(慕尼黑大学);National University of Singapore(新加坡国立大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Imperial College London(伦敦帝国理工学院)