ir_metadata：一种用于IR实验的可扩展元数据模式
ir_metadata: An Extensible Metadata Schema for IR Experiments
摘要：
信息检索（IR）社区有一个强大的传统，使计算工件和资源可供未来重用，允许实验结果的验证。除了实际的测试集合之外，底层的运行文件通常作为TREC、CLEF或NTCIR等会议的一部分托管在数据归档中。不幸的是，运行数据本身并没有提供关于底层实验的太多信息。例如，如果没有共享任务的网站或运行数据存档的上下文，单次运行文件就没有多大用处。在其他领域，如社会科学，用元数据注释研究数据是一种很好的做法。在这项工作中，我们介绍ir_metadata -一个可扩展的元数据模式的基础上PRIMAD模型的TREC运行文件。我们建议将元数据注释与PRIMAD对齐，PRIMAD考虑了可能影响再现性的计算实验组件。此外，我们还概述了元数据中应报告的重要组成部分和信息，并从文献中提供证据。为了证明这些元数据注释的有用性，我们在repro_eval中实现了支持再现性研究用例的元数据模式的新功能。此外，我们策划了一个数据集，其中包含从PRIMAD组件的不同实例化实验中获得的运行文件，并使用相应的元数据对其进行注释。在实验中，我们涵盖了由元数据识别并由PRIMAD分类的再现性实验。通过这项工作，我们使IR研究人员能够注释TREC运行文件，并进一步提高实验工件的重用价值。
英文摘要：
The information retrieval (IR) community has a strong tradition of making the computational artifacts and resources available for future reuse, allowing the validation of experimental results. Besides the actual test collections, the underlying run files are often hosted in data archives as part of conferences like TREC, CLEF, or NTCIR. Unfortunately, the run data itself does not provide much information about the underlying experiment. For instance, the single run file is not of much use without the context of the shared task's website or the run data archive. In other domains, like the social sciences, it is good practice to annotate research data with metadata. In this work, we introduce ir_metadata - an extensible metadata schema for TREC run files based on the PRIMAD model. We propose to align the metadata annotations to PRIMAD, which considers components of computational experiments that can affect reproducibility. Furthermore, we outline important components and information that should be reported in the metadata and give evidence from the literature. To demonstrate the usefulness of these metadata annotations, we implement new features in repro_eval that support the outlined metadata schema for the use case of reproducibility studies. Additionally, we curate a dataset with run files derived from experiments with different instantiations of PRIMAD components and annotate these with the corresponding metadata. In the experiments, we cover reproducibility experiments that are identified by the metadata and classified by PRIMAD. With this work, we enable IR researchers to annotate TREC run files and improve the reuse value of experimental artifacts even further.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
