从比较表自动生成早期排行榜
Automated Early Leaderboard Generation From Comparative Tables
摘要：
排行榜是一个表格，展示了解决特定科学问题的最佳竞争技术的表现得分。手动维护的排行榜需要时间才能出现，这会导致性能发现和有意义的比较延迟。这可能会延迟向非专家和从业人员传播最佳做法。关于代理的技术文件，我们提出了一个新的系统，自动发现和维护排行榜的形式之间的部分订单的文件，其中报告的性能为基础。原则上，排行榜取决于任务、数据集、其他实验设置和性能指标的选择。不同的度量标准之间往往也存在权衡。因此，排行榜发现不仅仅是准确提取性能数据并进行比较的问题。事实上，性能比较的噪声和不确定性水平如此之大，以至于可靠的传统提取是不可行的。我们通过使用论文中相对清晰、结构化的部分来减轻这些挑战，例如，性能表。我们提出了一种新的性能改进图的论文作为节点，边编码噪声性能比较信息提取表。每个单独的性能优势都是从包含其他论文引用的表格中提取的。这些提取类似于在不完全锦标赛中的“比赛”的（噪声）结果。我们提出了几种方法来排名这些嘈杂的“匹配”结果的文件。我们表明，我们的排名方案可以很好地再现各种手动策划的排行榜。使用广泛使用的国家的最先进的论文列表中的27个领域的计算机科学，我们证明了我们的系统产生非常可靠的排名。
英文摘要：
A leaderboard is a tabular presentation of performance scores of the best competing techniques that address a specific scientific problem. Manually maintained leaderboards take time to emerge, which induces a latency in performance discovery and meaningful comparison. This can delay dissemination of best practices to non-experts and practitioners. Regarding papers as proxies for techniques, we present a new system to automatically discover and maintain leaderboards in the form of partial orders between papers, based on performance reported therein. In principle, a leaderboard depends on the task, data set, other experimental settings, and the choice of performance metrics. Often there are also tradeoffs between different metrics. Thus, leaderboard discovery is not just a matter of accurately extracting performance numbers and comparing them. In fact, the levels of noise and uncertainty around performance comparisons are so large that reliable traditional extraction is infeasible. We mitigate these challenges by using relatively cleaner, structured parts of the papers, e.g., performance tables. We propose a novel performance improvement graph with papers as nodes, where edges encode noisy performance comparison information extracted from tables. Every individual performance edge is extracted from a table with citations to other papers. These extractions resemble (noisy) outcomes of 'matches' in an incomplete tournament. We propose several approaches to rank papers from these noisy 'match' outcomes. We show that our ranking scheme can reproduce various manually curated leaderboards very well. Using widely-used lists of state-of-the-art papers in 27 areas of Computer Science, we demonstrate that our system produces very reliable rankings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
