arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1807.06002cs.CYcs.PF
收藏

在Optil.io平台上实施的评估即服务架构和众包问题解决

Evaluation as a Service architecture and crowdsourced problems solving implemented in Optil.io platform

Szymon Wasik, Maciej Antczak, Jan Badura, Artur Laskowski

摘要:

算法的可靠性和可信性评估是一个具有挑战性的过程。首先,每种算法都有其优缺点,测试实例的选择对评估过程有很大的影响。其次,算法的测量性能高度依赖于测试环境架构,即,CPU型号、可用内存、缓存配置、操作系统内核,甚至编译标志。最后,通常很难将算法与其他研究人员编写的软件进行比较。评估即服务(Evaluation as a Service,EaaS)是一种云计算架构,它通过提供在线工具和测试实例来提高评估过程的可靠性。其中一个平台是Optil.io,它提供了几乎实时的定义问题、存储评估数据和评估研究人员提交的解决方案的可能性。在本文中,我们简要介绍了这个平台以及在其支持下组织的四个挑战。

英文摘要:

Reliable and trustworthy evaluation of algorithms is a challenging process. Firstly, each algorithm has its strengths and weaknesses, and the selection of test instances can significantly influence the assessment process. Secondly, the measured performance of the algorithm highly depends on the test environment architecture, i.e., CPU model, available memory, cache configuration, operating system's kernel, and even compilation flags. Finally, it is often difficult to compare algorithm with software prepared by other researchers. Evaluation as a Service (EaaS) is a cloud computing architecture that tries to make assessment process more reliable by providing online tools and test instances dedicated to the evaluation of algorithms. One of such platforms is Optil.io which gives the possibility to define problems, store evaluation data and evaluate solutions submitted by researchers in almost real time. In this paper, we briefly present this platform together with four challenges that were organized with its support.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑