Evaluating Temporal Graphs Built from Texts via Transitive Reduction
LIMSI-CNRS(法国国家科学研究中心计算机科学实验室);Univ. Paris-Sud(巴黎南大学);ALPAGE-INRIA(法国国家信息与自动化研究所ALPAGE项目组);Toulouse University(图卢兹大学);IRIT(图卢兹计算机科学研究所);Univ. Paul Sabatier(保罗·萨巴蒂埃大学)
Temporal information has been the focus of recent attention in information extraction, leading to some standardization effort, in particular for the task of relating events in a text. This task raises the problem of comparing two annotations of a given text, because relations between events in a story are intrinsically interdependent and cannot be evaluated separately. A proper evaluation measure is also crucial in the context of a machine learning approach to the problem. Finding a common comparison referent at the text level is not obvious, and we argue here in favor of a shift from event-based measures to measures on a unique textual object, a minimal underlying temporal graph, or more formally the transitive reduction of the graph of relations between event boundaries. We support it by an investigation of its properties on synthetic data and on a well-know temporal corpus.
Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
Barcelona Supercomputing Center, Spain(巴塞罗那超级计算中心);Universitat Politècnica de Catalunya, Spain(加泰罗尼亚理工大学);DFKI GmbH, Saarland Informatics Campus, Saarbrücken, Germany(DFKI GmbH萨尔兰信息技术校区)
Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
The Hong Kong Polytechnic University(香港理工大学);City University of Macau(澳门城市大学);Jilin University(吉林大学);Beihang University(北京航空航天大学);Jinan University(暨南大学)
FormalML: A Benchmark for Evaluating Formal Subgoal Completion in Machine Learning Theory
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);Department of Computer Science, ETH Zurich, Switzerland(瑞士苏黎世联邦理工学院计算机科学系);School of Computing and Mathematical Sciences, Birkbeck, University of London, UK(伦敦大学伯克贝克学院计算与数学科学学院)