arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 0804.3417astro-phcs.DC
收藏

鲁棒机器学习在Terascale天文数据集上的应用

Robust Machine Learning Applied to Terascale Astronomical Datasets

Nicholas M. Ball, Robert J. Brunner, Adam D. Myers

摘要:

我们介绍了UIUC Astronomy和NCSA之间的LCDM(宇宙学数据挖掘实验室; http://lcdm.astro.uiuc.edu)合作的最新结果,以部署超级计算集群资源和机器学习算法来挖掘万亿级天文数据集。这是天文学领域的一个新应用,因为我们正在使用这些资源进行数据挖掘,而不仅仅是进行模拟。通过NCSA网络增强数据到知识的改进实现,我们能够为斯隆数字巡天中的1亿多颗恒星和星系提供改进的分类,改进的距离测量,以及充分利用简单但功能强大的k-近邻算法。这项工作的一个驱动原则是,我们的方法应该是可扩展的,从当前的万亿次数据集到即将到来的千万亿次数据集和超越。我们讨论了迄今为止遇到的问题,以及过渡到千万亿次的进一步问题。特别是,除非实现必要的基础设施,否则磁盘I/O将成为一个主要的限制因素。

英文摘要:

We present recent results from the LCDM (Laboratory for Cosmological Data Mining; http://lcdm.astro.uiuc.edu) collaboration between UIUC Astronomy and NCSA to deploy supercomputing cluster resources and machine learning algorithms for the mining of terascale astronomical datasets. This is a novel application in the field of astronomy, because we are using such resources for data mining, and not just performing simulations. Via a modified implementation of the NCSA cyberenvironment Data-to-Knowledge, we are able to provide improved classifications for over 100 million stars and galaxies in the Sloan Digital Sky Survey, improved distance measures, and a full exploitation of the simple but powerful k-nearest neighbor algorithm. A driving principle of this work is that our methods should be extensible from current terascale datasets to upcoming petascale datasets and beyond. We discuss issues encountered to-date, and further issues for the transition to petascale. In particular, disk I/O will become a major limiting factor unless the necessary infrastructure is implemented.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑