基于开放评审的（ORB）数据集：迈向高能物理论文与实验提案的自动评估
The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics
AI总结：
本文构建包含 3.6 万余篇论文、8.9 万余条评审与决定的 ORB 数据集及自动更新基础设施，并通过基于文本嵌入的 NLP 初步实验验证论文录用与评分预测。
摘要：
随着开放科学方法在研究中变得日益重要，向开放科学论文评审的演进正在对科学界产生影响。然而，目前缺乏用于开展与该主题相关研究活动的公开可用资源，因为只有数量有限的期刊和会议允许相关方访问其评审过程。本文介绍了一个全新的、综合性的基于开放评审的数据集（Open Review-Based dataset，ORB）；它包含经过整理的 36,000 多篇科学论文，以及这些论文的 89,000 多条评审意见和最终决定。我们从两个来源收集这些信息：OpenReview.net 和 SciPost.org 网站。不过，鉴于该领域具有易变的性质，我们为补充 ORB 数据集而引入的软件基础设施被设计为可在未来容纳更多资源。ORB 的交付成果包括：（1）用于将文档数据和元数据转换为结构化高层表示的 Python 代码（接口与实现）；（2）用于促进从既定来源自动更新的 ETL（Extract, Transform，抽取、转换、加载）流程；以及（3）表示结构化数据的数据文件。本文介绍了我们的数据架构、所收集数据的概览以及相关统计信息。为便于说明，我们还讨论了基于自然语言处理的初步实验，这些实验旨在预测：（1）基于论文文本嵌入的论文录用情况；（2）同样由嵌入推断出的评分统计信息。我们相信 ORB 为关注开放科学和开放评审的研究人员提供了宝贵资源，而我们的实现也简化了这些数据在进一步分析和实验中的使用。我们计划随着该领域的成熟持续更新 ORB，并引入更适合高能物理等专门科学领域的新资源。
英文摘要：
With the Open Science approach becoming important for research, the evolution towards open scientific-paper reviews is making an impact on the scientific community. However, there is a lack of publicly available resources for conducting research activities related to this subject, as only a limited number of journals and conferences currently allow access to their review process for interested parties. In this paper, we introduce the new comprehensive Open Review-Based dataset (ORB); it includes a curated list of more than 36,000 scientific papers with their more than 89,000 reviews and final decisions. We gather this information from two sources: the OpenReview.net and SciPost.org websites. However, given the volatile nature of this domain, the software infrastructure that we introduce to supplement the ORB dataset is designed to accommodate additional resources in the future. The ORB deliverables include (1) Python code (interfaces and implementations) to translate document data and metadata into a structured and high-level representation, (2) an ETL process (Extract, Transform, Load) to facilitate the automatic updates from defined sources and (3) data files representing the structured data. The paper presents our data architecture and an overview of the collected data along with relevant statistics. For illustration purposes, we also discuss preliminary Natural-Language-Processing-based experiments that aim to predict (1) papers' acceptance based on their textual embeddings, and (2) grading statistics inferred from embeddings as well. We believe ORB provides a valuable resource for researchers interested in open science and review, with our implementation easing the use of this data for further analysis and experimentation. We plan to update ORB as the field matures as well as introduce new resources even more fitted to dedicated scientific domains such as High-Energy Physics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
