Constructing a timeline requires identifying the chronological order of events in an article. In prior timeline construction datasets, temporal orders are typically annotated by either event-to-time anchoring or event-to-event pairwise ordering, both of which suffer from missing temporal information. To mitigate the issue, we develop a new evaluation dataset, TimeSET, consisting of single-document timelines with document-level order annotation. TimeSET features saliency-based event selection and partial ordering, which enable a practical annotation workload. Aiming to build better automatic timeline construction systems, we propose a novel evaluation framework to compare multiple task formulations with TimeSET by prompting open LLMs, i.e., Llama 2 and Flan-T5. Considering that identifying temporal orders of events is a core subtask in timeline construction, we further benchmark open LLMs on existing event temporal ordering datasets to gain a robust understanding of their capabilities. Our experiments show that (1) NLI formulation with Flan-T5 demonstrates a strong performance among others, while (2) timeline construction and event temporal ordering are still challenging tasks for few-shot LLMs. Our code and data are available at https://github.com/kimihiroh/timeset.
DTELS: Towards Dynamic Granularity of Timeline Summarization
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Who's important? -- SUnSET: Synergistic Understanding of Stakeholder, Events and Time for Timeline Generation
National University of Singapore(国立新加坡大学);A*STAR Institute of High Performance Computing(A*STAR高性能计算研究所);Celia Research Team(Celia研究团队);Norbert Wiener Research Center, Huawei International(诺伯特·维纳研究中心,华为国际)
ProtoMed-LLM: An Automatic Evaluation Framework for Large Language Models in Medical Protocol Formulation
University of Texas at Austin(得克萨斯大学奥斯汀分校);Ulsan National Institute of Science and Technology(蔚山国家科学技术研究院);Korea Institute of Science and Technology (KIST) Europe(韩国科学技术研究院欧洲分院)
HalluScore: Large Language Model Hallucination Question Answering Benchmark
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals(国王法赫德石油与矿物大学信息与计算机科学系);SDAIA-KFUPM Joint Research Center for Artificial Intelligence(SDAIA-KFUPM人工智能联合研究中心)
Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Design
School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);School of Engineering and Computer Science, Victoria University of Wellington(惠灵顿维多利亚大学工程与计算机科学学院);School of Engineering, Westlake University(西湖大学工学院)
Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs
Technische Universität Berlin(柏林工业大学);Deutsches Forschungszentrum für Künstliche Intelligenz(德国人工智能研究中心);National Institute of Informatics(信息学研究所);Harz University of Applied Sciences(哈尔茨应用技术大学)
Reasoning Factual Knowledge in Structured Data with Large Language Models
Hong Kong Polytechnic University(香港理工大学);University Technology Sydney(悉尼科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Soochow University(苏州大学);The Education University of Hong Kong(香港教育大学)
Do "New Snow Tablets" Contain Snow? Large Language Models Over-Rely on Names to Identify Ingredients of Chinese Drugs
Liaoning University(辽宁大学);University of Queensland(昆士兰大学);National University of Singapore(新加坡国立大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Merced(加利福尼亚大学默塞德分校)
Commonsense Knowledge Editing Based on Free-Text in LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
Reconstructing Sepsis Trajectories from Clinical Case Reports using LLMs: the Textual Time Series Corpus for Sepsis
Machine Learning Department and Heinz College Carnegie Mellon University(卡内基梅隆大学机器学习系和海恩兹学院);National Library of Medicine National Institutes of Health(美国国家医学图书馆)
Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Columbia University(哥伦比亚大学);RWTH Aachen(亚琛工业大学);University of Michigan(密歇根大学);ETH Zürich(苏黎世联邦理工学院);Max Planck Institute(马克斯·普朗克研究所)
FormuEvo: LLM-Guided Evolution for Discovering Solver-Efficient Mixed-Integer Programming Formulations
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院);Tsinghua University(清华大学)
Improving LLMs via Validator-to-Generator Alignment
Department of Computer Science, The University of Texas at Austin(德克萨斯大学奥斯汀分校计算机科学系);Departments of Linguistics and Computer Science, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校语言学与计算机科学系);Department of Computer Science & Center for Data Science, New York University(纽约大学计算机科学系及数据科学中心)
KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation
Beijing University of Posts and Telecommunications(北京邮电大学);Tongji University(同济大学);Multimodal Art Projection Research Community(多模态艺术投影研究社区);Duke University(杜克大学);ZTE Corporation(中兴通讯股份有限公司)
ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making
School of Data Science, Fudan University(复旦大学数据科学学院);Obstetrics & Gynecology Hospital of Fudan University(复旦大学附属妇产科医院);Shanghai Innovation Institute(上海创新研究院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院)