使用Vaex对十亿个对象进行交互式(统计)可视化和探索
Interactive (statistical) visualisation and exploration of a billion objects with Vaex
摘要:
随着新目录的到来,如盖亚DR 1,包含超过10亿个对象,需要新的方法来处理和可视化这些数据量。在可视化中,一个问题是数据点的数量可能会变得如此之大,以至于散点图变得混乱。另一个问题是,对于超过10亿个对象,如果想要在一秒钟内处理它们,每个对象只有几个cpu周期可用,这使得传统的绘制字形的方法不可行。相反,我们表明,通过计算一个规则的(N维)网格上的统计数据,十亿个对象的可视化可以在一秒钟内完成一个现代的台式电脑。这是通过使用hdf 5文件的内存映射和一个简单的装箱算法来实现的,它们是Python库vaex的一部分。这使得有效的探索或大型数据集交互,使科学探索大型目录可行。Vaex是一个Python库,它也可以很好地集成在Python/Numpy/Astropy/matplotlib堆栈中。在此之上构建的是vaex应用程序,它允许交互式探索和可视化。开发vaex的动机是盖亚卫星的目录,然而,vaex也可以用于SPH或N体模拟,任何其他(未来)目录,如SDSS,Pan-STARRS,LSST,WISE,2 MASS等或其他表格数据。vaex的主页是http://vaex.astro.rug.nl。
英文摘要:
With new catalogues arriving such as the Gaia DR1, containing more than a billion objects, new methods of handling and visualizing these data volumes are needed. In visualization, one problem is that the number of datapoints can become so large, that a scatter plot becomes cluttered. Another problem is that with over a billion objects, only a few cpu cycles are available per object if one wants to process them within a second, making traditional methods by rendering glyphs not viable. Instead, we show that by calculating statistics on a regular (N-dimensional) grid, visualizations of a billion objects can be done within a second on a modern desktop computer. This is achieved using memory mapping of hdf5 files together with a simple binning algorithm, which are part of a Python library called vaex. This enables efficient exploration or large datasets interactively, making science exploration of large catalogues feasible. Vaex is a Python library, which also integrates well in the Jupyter/Numpy/Astropy/matplotlib stack. Build on top of this is the vaex application, which allows for interactive exploration and visualization. The motivation for developing vaex is the catalogue of the Gaia satellite, however, vaex can also be used on SPH or N-body simulations, any other (future) catalogues such as SDSS, Pan-STARRS, LSST, WISE, 2MASS, etc. or other tabular data. The homepage for vaex is http://vaex.astro.rug.nl.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。