Classification is valuable and necessary in spectral analysis, especially for data-driven mining. Along with the rapid development of spectral surveys, a variety of classification techniques have been successfully applied to astronomical data processing. However, it is difficult to select an appropriate classification method in practical scenarios due to the different algorithmic ideas and data characteristics. Here, we present the second work in the data mining series - a review of spectral classification techniques. This work also consists of three parts: a systematic overview of current literature, experimental analyses of commonly used classification algorithms and source codes used in this paper. Firstly, we carefully investigate the current classification methods in astronomical literature and organize these methods into ten types based on their algorithmic ideas. For each type of algorithm, the analysis is organized from the following three perspectives. (1) their current applications and usage frequencies in spectral classification are summarized; (2) their basic ideas are introduced and preliminarily analysed; (3) the advantages and caveats of each type of algorithm are discussed. Secondly, the classification performance of different algorithms on the unified data sets is analysed. Experimental data are selected from the LAMOST survey and SDSS survey. Six groups of spectral data sets are designed from data characteristics, data qualities and data volumes to examine the performance of these algorithms. Then the scores of nine basic algorithms are shown and discussed in the experimental analysis. Finally, nine basic algorithms source codes written in python and manuals for usage and improvement are provided.
Astronomy, as many other scientific disciplines, is facing a true data deluge which is bound to change both the praxis and the methodology of every day research work. The emerging field of astroinformatics, while on the one end appears crucial to face the technological challenges, on the other is opening new exciting perspectives for new astronomical discoveries through the implementation of advanced data mining procedures. The complexity of astronomical data and the variety of scientific problems, however, call for innovative algorithms and methods as well as for an extreme usage of ICT techn…
Clustering is an effective tool for astronomical spectral analysis, to mine clustering patterns among data. With the implementation of large sky surveys, many clustering methods have been applied to tackle spectroscopic and photometric data effectively and automatically. Meanwhile, the performance of clustering methods under different data characteristics varies greatly. With the aim of summarizing astronomical spectral clustering algorithms and laying the foundation for further research, this work gives a review of clustering methods applied to astronomical spectra data in three parts. First,…
The exponential growth of astronomical data collected by both ground based and space borne instruments has fostered the growth of Astroinformatics: a new discipline laying at the intersection between astronomy, applied computer science, and information and computation (ICT) technologies. At the very heart of Astroinformatics is a complex set of methodologies usually called Data Mining (DM) or Knowledge Discovery in Data Bases (KDD). In the astronomical domain, DM/KDD are still in a very early usage stage, even though new methods and tools are being continuously deployed in order to cope with t…
This is a companion Focus Demonstration article to the CANFAR+Skytree poster (Ball 2012), demonstrating the usage of the Skytree machine learning software on the Canadian Advanced Network for Astronomical Research (CANFAR) cloud computing system. CANFAR+Skytree is the world's first cloud computing system for data mining in astronomy.
In 2020, ~60PB of archived data will be accessible to the astronomers. But to analyze such a paramount data will be a challenging task. This is basically due to the computational model used to download the data from complex geographically distributed archives to a central site and then analyzing it in the local systems. Because the data has to be downloaded to the central site, the network BW limitation will be a hindrance for the scientific discoveries. Also analyzing this PB-scale on local machines in a centralized manner is challenging. In this virtual observatory is a step towards this pro…
The world astronomical image archives represent huge opportunities to time-domain astronomy sciences and other hot topics such as space defense, and astronomical observatories should improve this wealth and make it more accessible in the big data era. In 2010 we introduced the Mega-Archive database and the Mega-Precovery server for data mining images containing Solar system bodies, with focus on near Earth asteroids (NEAs). This paper presents the improvements and introduces some new related data mining tools developed during the last five years. Currently, the Mega-Archive has indexed 15 mill…
Astronomical data are gathered through a very large number of heterogeneous techniques and stored in very diversified and often incompatible data repositories. Moreover in the e-science environment, it is needed to integrate services across distributed, heterogeneous, dynamic "virtual organizations" formed by different resources within a single enterprise and/or external resource sharing and service provider relationships. The DAME/VONeural project, run jointly by the University Federico II, INAF (National Institute of Astrophysics) Astronomical Observatories of Napoli and the California Insti…
We propose and preliminarily implement a data-mining based platform to assist experts to inspect the increasing amount of spectra with low signal to noise ratio (SNR) generated by large sky surveys. The platform includes three layers: data-mining layer, data-node layer and expert layer. It is similar to the GalaxyZoo project and VO-compatible. The preliminary experiment suggests that this platform can play an effective role in managing the spectra and assisting the experts to inspect a large number of spectra with low SNR.
A trade-off between speed and information controls our understanding of astronomical objects. Fast-to-acquire photometric observations provide global properties, while costly and time-consuming spectroscopic measurements enable a better understanding of the physics governing their evolution. Here, we tackle this problem by generating spectra directly from photometry, through which we obtain an estimate of their intricacies from easily acquired images. This is done by using multi-modal conditional diffusion models, where the best out of the generated spectra is selected with a contrastive netwo…
We present an interactive IDL program for viewing and analyzing astronomical spectra in the context of modern imaging surveys. SpecPro's interactive design lets the user simultaneously view spectroscopic, photometric, and imaging data, allowing for rapid object classification and redshift determination. The spectroscopic redshift can be determined with automated cross-correlation against a variety of spectral templates or by overlaying common emission and absorption features on the 1-D and 2-D spectra. Stamp images as well as the spectral energy distribution (SED) of a source can be displayed …
Mid-infrared (IR) array detectors have been used for astronomical observations in space. However, the uniformities of their spectral response curves have not been investigated in detail, the understanding of which is important for spectroscopic observations using large array formats. We characterize the spectral responses of all the pixels in IR array detectors using a Fourier transform infrared spectrometer and cryogenic optics for measurements at high signal-to-noise ratios. We measured the spectral responses of the Si:As impurity band conduction (IBC) array, a flight back-up detector for AK…
We present a new solution for the dispersive element in astronomical spectrographs, which in many cases can provide an upgrade path to enhance the spectral resolution of existing moderate-resolution reflection-grating spectrographs. We demonstrate that in the case of LRIS-R at the Keck 1 Telescope a spectral resolution of 18,000 can be achieved with reasonable throughput under good seeing conditions.
With the application of advanced astronomical technologies, equipments and methods all over the world, astronomy covers from radio, infrared, visible light, ultraviolet, X-ray and gamma ray band, and enters into the era of full wavelength astronomy. How to effectively integrate data from different ground- and space-based observation equipments, different observers, different bands, different observation time, requires the data fusion technology. In this paper we introduce the cross-match tool that is developed by the Python language and based on the PostgreSQL database and uses Q3C as the core…
We present a powerful new algorithm that combines both spatial information (event locations and the point spread function) and spectral information (photon energies) to separate photons from overlapping sources. We use Bayesian statistical methods to simultaneously infer the number of overlapping sources, to probabilistically separate the photons among the sources, and to fit the parameters describing the individual sources. Using the Bayesian joint posterior distribution, we are able to coherently quantify the uncertainties associated with all these parameters. The advantages of combining spa…
Astronomy is undergoing through a methodological revolution triggered by an unprecedented wealth of complex and accurate data. The new panchromatic, synoptic sky surveys require advanced tools for discovering patterns and trends hidden behind data which are both complex and of high dimensionality. We present DAMEWARE (DAta Mining & Exploration Web Application REsource): a general purpose, web-based, distributed data mining environment developed for the exploration of large datasets, and finely tuned for astronomical applications. By means of graphical user interfaces, it allows the user to per…
In this contribution, we present WebPlotDigitizer, a polyvalent and free software developed to facilitate easy and accurate data extraction from a variety of plot types. We describe the numerous features of this numerical tool and present its relevance when applied to astrophysical archival research. We exploit WebPlotDigitizer to extract ultraviolet spectropolarimetric spectra from old publications that used the Hubble Space Telescope, Lick Observatory 3m Shane telescope and Astro-2 mission to observe the Seyfert-2 AGN NGC 1068. By doing so, we compile all the existing ultraviolet polarimetri…
Besides new observations, mining old photographic plates and CCD image archives represents an opportunity to recover and secure newly discovered asteroids, also to improve the orbits of Near Earth Asteroids (NEAs), Potentially Hazardous Asteroids (PHAs) and Virtual Impactors (VIs). These are the main research aims of the EURONEAR network. As stated by the IAU, the vast collection of image archives stored worldwide is still insufficiently explored, and could be mined for known NEAs and other asteroids appearing occasionally in their fields. This data mining could be eased using a server to sear…
We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining proce…
Many estimation problems in astrophysics are highly complex, with high-dimensional, non-standard data objects (e.g., images, spectra, entire distributions, etc.) that are not amenable to formal statistical analysis. To utilize such data and make accurate inferences, it is crucial to transform the data into a simpler, reduced form. Spectral kernel methods are non-linear data transformation methods that efficiently reveal the underlying geometry of observable data. Here we focus on one particular technique: diffusion maps or more generally spectral connectivity analysis (SCA). We give examples o…
We present an application of unsupervised machine learning - the self-organised map (SOM) - as a tool for visualising, exploring and mining the catalogues of large astronomical surveys. Self-organisation culminates in a low-resolution representation of the 'topology' of a parameter volume, and this can be exploited in various ways pertinent to astronomy. Using data from the Cosmological Evolution Survey (COSMOS), we demonstrate two key astronomical applications of the SOM: (i) object classification and selection, using the example of galaxies with active galactic nuclei as a demonstration, and…
Traditional analysis techniques may not be sufficient for astronomers to make the best use of the data sets that current and future instruments, such as the Square Kilometre Array and its Pathfinders, will produce. By utilizing the incredible pattern-recognition ability of the human mind, scientific visualization provides an excellent opportunity for astronomers to gain valuable new insight and understanding of their data, particularly when used interactively in 3D. The goal of our work is to establish the feasibility of a real-time 3D monitoring system for data going into the Australian SKA P…
Observatories are complex scientific and technical institutions serving diverse users and purposes. Their telescopes, instruments, software, and human resources engage in interwoven workflows over a broad range of timescales. These workflows have been tuned to be responsive to concepts of observatory operations that were applicable when various assets were commissioned, years or decades in the past. The astronomical community is entering an era of rapid change increasingly characterized by large time domain surveys, robotic telescopes and automated infrastructures, and - most significantly - o…
The main purpose of this article is to make Astronomers aware that Searches for Extraterrestrial Intelligence can be carried out by analyzing standard astronomical spectra, including those they already have taken. Simplicity is the outstanding advantage of a search in spectra. The spectra can be analyzed by simple eye inspection or a few lines of code that uses Fourier transform software. Theory, confirmed by published experiments, shows that periodic signals in spectra can be easily generated by sending light pulses separated by constant time intervals. While part of this article, like all ar…
We present a technique to synthesise telluric absorption and emission features both for in-situ wavelength calibration and for their removal from astronomical spectra. While the presented technique is applicable for a wide variety of optical and infrared spectra, we concentrate in this paper on selected high-resolution near-infrared spectra obtained with the CRIRES spectrograph to demonstrate its performance and limitation. We find that synthetic spectra reproduce telluric absorption features to about 2%, even close to saturated line cores. Thus, synthetic telluric spectra could be used to rep…
Nowadays, many scientific areas share the same need of being able to deal with massive and distributed datasets and to perform on them complex knowledge extraction tasks. This simple consideration is behind the international efforts to build virtual organizations such as, for instance, the Virtual Observatory (VObs). DAME (DAta Mining & Exploration) is an innovative, general purpose, Web-based, VObs compliant, distributed data mining infrastructure specialized in Massive Data Sets exploration with machine learning methods. Initially fine tuned to deal with astronomical data only, DAME has evol…
radio-astro-tools: linking radio astronomical data to the astronomical Python ecosystem
Dominion Radio Astrophysical Observatory, Herzberg Astronomy & Astrophysics, National Research Council Canada（加拿大国家研究委员会赫茨伯格天文学与天体物理学多明尼奥射电天文台）；Department of Astronomy, University of Florida（佛罗里达大学天文学系）；Aperio Software（阿佩里奥软件公司）；Dept. of Physics, University of Alberta（阿尔伯塔大学物理系）；Department of Physics and Astronomy, Haverford College（哈弗福德学院物理与天文学系）；Max-Planck-Institut für Astronomie（马克斯·普朗克天文学研究所）；National Radio Astronomy Observatory（美国国家射电天文台）；Department of Astronomy, The Ohio State University（俄亥俄州立大学天文学系）；Center for Cosmology and Astroparticle Physics (CCAPP)（宇宙学与基本粒子物理中心）；Univ. Grenoble Alpes, CNRS, IPAG（格勒诺布尔阿尔卑斯大学，法国国家科学研究中心，IPAG）
We describe the application of data mining algorithms to research problems in astronomy. We posit that data mining has always been fundamental to astronomical research, since data mining is the basis of evidence-based discovery, including classification, clustering, and novelty discovery. These algorithms represent a major set of computational tools for discovery in large databases, which will be increasingly essential in the era of data-intensive astronomy. Historical examples of data mining in astronomy are reviewed, followed by a discussion of one of the largest data-producing projects anti…
In this paper we present a general forward fitting method for multichannel image restoration based on regularized chi2. We introduce separable regularizations that account for the dynamic of the model and take advantage of the continuities present in the data, leaving only two hyper-parameters to tune. We illustrate a practical implementation of this method in the context of host galaxy subtraction for the Nearby SuperNova factory. We show that the image restoration obtained fulfills the stringent requirements on bias and photometricity needed by this program. The reconstruction yields sub-per…
Context. K-means is a clustering algorithm that has been used to classify large datasets in astronomical databases. It is an unsupervised method, able to cope very different types of problems. Aims. We check whether a variant of the algorithm called single-pass k-means can be used as a fast alternative to the traditional k-means. Methods. The execution time of the two algorithms are compared when classifying subsets drawn from the SDSS-DR7 catalog of galaxy spectra. Results. Single-pass k-means turn out to be between 20 % and 40 % faster than k-means and provide statistically equivalent classi…
INAF – Osservatorio Astronomico di Capodimonte（意大利国家天体物理研究所卡波迪蒙特天文台）；Università degli Studi Federico II（费德里科二世大学）；INAF – Osservatorio Astronomico di Trieste（的里雅斯特天文台）
Modern scientific data mainly consist of huge datasets gathered by a very large number of techniques and stored in very diversified and often incompatible data repositories. More in general, in the e-science environment, it is considered as a critical and urgent requirement to integrate services across distributed, heterogeneous, dynamic "virtual organizations" formed by different resources within a single enterprise. In the last decade, Astronomy has become an immensely data rich field due to the evolution of detectors (plates to digital to mosaics), telescopes and space instruments. The Virt…
The new wide-field radio telescopes, such as: ASKAP, MWA, LOFAR, eVLA and SKA; will produce spectral-imaging data-cubes (SIDC) of unprecedented size -- in the order of hundreds of Petabytes. Servicing such data as images to the end-user in a traditional manner and formats is likely going to encounter significant performance fallbacks. We discuss the requirements for extremely large SIDCs, and in this light we analyse the applicability of the approach taken in the JPEG2000 (ISO/IEC 15444) standards. We argue the case for the adaptation of contemporary industry standards and technologies vs the …
We investigate the performance of the generalized Spectral Kurtosis (SK) estimator in detecting and discriminating natural and artificial very short duration transients in the 2-bit sampling time domain Very-Long-Baseline Interferometry (VLBI) data. We demonstrate that, while both types of transients may be efficiently detected, their natural or artificial nature cannot be distinguished if only a time domain SK analysis is performed. However, these two types of transients become distinguishable from each other in the spectral domain, after a 32-bit FFT operation is performed on the 2-bit time …
Applying Vision Transformers on Spectral Analysis of Astronomical Objects
Harvard Extension School（哈佛大学延伸学校）；Harvard University（哈佛大学）；John A. Paulson School of Engineering and Applied Science（约翰·A·保罗森工程与应用科学学院）；Department of Computer Science（计算机科学系）；Center for Data and Artificial Intelligence（数据与人工智能中心）；Millennium Institute of Astrophysics (MAS)（千年天体物理研究所 (MAS)）；Millennium Nucleus for Galaxies (MINGAL)（银河千年核 (MINGAL)）；Heidelberg Institute for Theoretical Studies（海德堡理论研究所）
We present the Spectroscopic Classification of Astronomical Transients (SCAT) survey, which is dedicated to spectrophotometric observations of transient objects such as supernovae and tidal disruption events. SCAT uses the SuperNova Integral-Field Spectrograph (SNIFS) on the University of Hawai'i 2.2-meter (UH2.2m) telescope. SNIFS was designed specifically for accurate transient spectrophotometry, including absolute flux calibration and host-galaxy removal. We describe the data reduction and calibration pipeline including spectral extraction, telluric correction, atmospheric characterization,…
Thorium hollow cathode lamps (HCLs) are used as frequency calibrators for many high resolution astronomical spectrographs, some of which aim for Doppler precision at the 1 m/s level. We aim to determine the most suitable combination of elements (Th or U, Ar or Ne) for wavelength calibration of astronomical spectrographs, to characterize differences between similar HCLs, and to provide a new U line-list. We record high resolution spectra of different HCLs using a Fourier transform spectrograph: (i) U-Ne, U-Ar, Th-Ne, and Th-Ar lamps in the spectral range from 500 to 1000 nm and U-Ne and U-Ar fr…
Millennium Institute of Astrophysics（千禧天体物理研究所）；Universidad de Chile（智利大学）；Harvard University（哈佛大学）；University of Florida（佛罗里达大学）；Universidad de los Andes（智利洛斯安第斯大学）
Time-domain astronomy (TDA) is facing a paradigm shift caused by the exponential growth of the sample size, data complexity and data generation rates of new astronomical sky surveys. For example, the Large Synoptic Survey Telescope (LSST), which will begin operations in northern Chile in 2022, will generate a nearly 150 Petabyte imaging dataset of the southern hemisphere sky. The LSST will stream data at rates of 2 Terabytes per hour, effectively capturing an unprecedented movie of the sky. The LSST is expected not only to improve our understanding of time-varying astrophysical objects, but al…
University of Toulouse（图卢兹大学）；IRIT/INP-ENSEEIHT（图卢兹计算机研究所/国立图卢兹应用科学学院-国立高等电力技术、电子学、计算机、水力学与电信学院）；CNRS（法国国家科学研究中心）；Institut Universitaire de France (IUF)（法国大学研究院）；ISAE-SUPAERO（法国高等航空航天学院）；Institut de Recherche en Astrophysique et Planétologie (IRAP)（天体物理与行星学研究所）
Hyperspectral imaging has become a significant source of valuable data for astronomers over the past decades. Current instrumental and observing time constraints allow direct acquisition of multispectral images, with high spatial but low spectral resolution, and hyperspectral images, with low spatial but high spectral resolution. To enhance scientific interpretation of the data, we propose a data fusion method which combines the benefits of each image to recover a high spatio-spectral resolution datacube. The proposed inverse problem accounts for the specificities of astronomical instruments, …
This elementary review covers the basics of working with astronomical data, notably with images, spectra and higher-level (catalog) data. The basic concepts and tools are presented using both application software (DS9 and TOPCAT) and Python. The level of presentation is suitable for undergraduate students, but should also be accessible to advanced high school students.
Infrared spectroscopic observations have established the presence of hydrocarbon ices on Pluto and other TNOs, but the abundances of such molecules cannot be deduced without accurate optical constants (n, k) and reference spectra. In this paper we present our recent measurements of near- and mid-infrared optical constants for ethane (C$_2$H$_6$) and ethylene (C$_2$H$_4$) in multiple ice phases and at multiple temperatures. As in our recent work on acetylene (C$_2$H$_2$), we also report new measurements of the index of refraction of each ice at 670 nm. Comparisons are made to earlier work where…
Theoretical spectral energy distributions (SEDs) of white dwarfs provide a powerful tool for cross-calibration and sensitivity control of instruments from the far infrared to the X-ray energy range. Such SEDs can be calculated from fully metal-line blanketed NLTE model-atmospheres that are e.g. computed by the Tuebingen NLTE model-atmosphere package (TMAP) that has arrived at a high level of sophistication. TMAP was successfully employed for the reliable spectral analysis of many hot, compact post-AGB stars. High-quality stellar spectra obtained over a wide energy range establish a data base w…
One promising application of photonics to astronomical instrumentation is the miniaturization of near-infrared (NIR) spectrometers for large ground- and space-based astronomical telescopes. Here we present new results from our effort to fabricate arrayed waveguide grating (AWG) spectrometers for astronomical applications entirely in-house. Our latest devices have a peak overall throughput of ~23%, a spectral resolving power ($λ/δλ$) of ~1300, and cover the entire H band (1450-1650 nm) for Transverse Electric (TE) polarization. These AWGs use a silica-on-silicon platform with a very thin layer …
University College London（伦敦大学学院）；Columbia University（哥伦比亚大学）；Flatiron Institute（弗莱提荣研究所）；Leiden University（莱顿大学）
Modern astronomical surveys are observing spectral data for millions of stars. These spectra contain chemical information that can be used to trace the Galaxy's formation and chemical enrichment history. However, extracting the information from spectra, and making precise and accurate chemical abundance measurements are challenging. Here, we present a data-driven method for isolating the chemical factors of variation in stellar spectra from those of other parameters (i.e. \teff, \logg, \feh). This enables us to build a spectral projection for each star with these parameters removed. We do this…
We describe and characterize a 25 GHz laser frequency comb based on a cavity-filtered erbium fiber mode-locked laser. The comb provides a uniform array of optical frequencies spanning 1450 nm to 1700 nm, and is stabilized by use of a global positioning system referenced atomic clock. This comb was deployed at the 9.2 m Hobby-Eberly telescope at the McDonald Observatory where it was used as a radial velocity calibration source for the fiber-fed Pathfinder near-infrared spectrograph. Stellar targets were observed in three echelle orders over four nights, and radial velocity precision of \sim10 m…
With the aim of utilizing arrayed waveguide gratings for multi-object spectroscopy in the field of astronomy, we outline several ways in which standard telecommunications grade chips should be modified. In particular, by removing the parabolic-horn taper or multimode interference coupler, and injecting with an optical fiber directly, the resolving power was increased threefold from 2400 \pm 200 (spectral resolution of 0.63 \pm 0.2 nm) to 7000 \pm 700 (0.22 \pm 0.02 nm) while attaining a throughput of 77 \pm 5%. More importantly, the removal of the taper enabled simultaneous off-axis injection …
We present the pulsar_spectra software repository, an open-source pulsar flux density catalogue and automated spectral fitting software that finds the best spectral model and produces publication-quality plots. The Python-based software includes features that enable users in the astronomical community to add newly published spectral measurements to the catalogue as they become available. The spectral fitting software is an implementation of the method described in Jankowski et al. (2018) which uses robust statistical methods to decide on the best-fitting model for individual pulsar spectra. pu…
The analysis of individual X-ray sources that appear in a crowded field can easily be compromised by the misallocation of recorded events to their originating sources. Even with a small number of sources, that nonetheless have overlapping point spread functions, the allocation of events to sources is a complex task that is subject to uncertainty. We develop a Bayesian method designed to sift high-energy photon events from multiple sources with overlapping point spread functions, leveraging the differences in their spatial, spectral, and temporal signatures. The method probabilistically assigns…
