Trend Filtering -- I. A Modern Statistical Tool for Time-Domain Astronomy and Astronomical Spectroscopy
Collin A. Politsch, Jessi Cisewski-Kehe, Rupert A. C. Croft, Larry Wasserman
备注：
Part 1 of 2; 15 pages, 3 figures
发表信息：
Trend filtering -- I. A modern statistical tool for time-domain astronomy and astronomical spectroscopy, Monthly Notices of the Royal Astronomical Society, Volume 492, Issue 3, March 2020, Pages 4005 - 4018
The problem of denoising a one-dimensional signal possessing varying degrees of smoothness is ubiquitous in time-domain astronomy and astronomical spectroscopy. For example, in the time domain, an astronomical object may exhibit a smoothly varying intensity that is occasionally interrupted by abrupt dips or spikes. Likewise, in the spectroscopic setting, a noiseless spectrum typically contains intervals of relative smoothness mixed with localized higher frequency components such as emission peaks and absorption lines. In this work, we present trend filtering, a modern nonparametric statistical tool that yields significant improvements in this broad problem space of denoising $spatially$ $heterogeneous$ signals. When the underlying signal is spatially heterogeneous, trend filtering is superior to any statistical estimator that is a linear combination of the observed data---including kernel smoothers, LOESS, smoothing splines, Gaussian process regression, and many other popular methods. Furthermore, the trend filtering estimate can be computed with practical and scalable efficiency via a specialized convex optimization algorithm, e.g. handling sample sizes of $n\gtrsim10^7$ within a few minutes. In a companion paper, we explicitly demonstrate the broad utility of trend filtering to observational astronomy by carrying out a diverse set of spectroscopic and time-domain analyses.
Why study astronomy, why teach astronomy? We give answers to these fundamental questions based on our experience with the Astronomical Camp "Beli Brezi" (White Aspens; Kardzhali, Bulgaria). It has been a place for teaching astronomy to high schools kids for nearly half a century. We describe shortly the history of the camp and draw some conclusions based on nearly five decades of experience. Major among them is that the camp has gone further than just distributing astronomical knowledge - while this is an important and worthy task, the main achievement has been the cultivation of critical thin…
Trend filtering---first introduced into the astronomical literature in Paper I of this series---is a state-of-the-art statistical tool for denoising one-dimensional signals that possess varying degrees of smoothness. In this work, we demonstrate the broad utility of trend filtering to observational astronomy by discussing how it can contribute to a variety of spectroscopic and time-domain studies. The observations we discuss are (1) the Lyman-$α$ forest of quasar spectra; (2) more general spectroscopy of quasars, galaxies, and stars; (3) stellar light curves with planetary transits; (4) eclips…
We introduce \emph{Astronomy and Computing}, a new journal for the growing population of people working in the domain where astronomy overlaps with computer science and information technology. The journal aims to provide a new communication channel within that community, which is not well served by current journals, and to help secure recognition of its true importance within modern astronomy. In this inaugural editorial, we describe the rationale for creating the journal, outline its scope and ambitions, and seek input from the community in defining in detail how the journal should work towar…
Here we report on the Women in Astronomy Workshop (http://asawomeninastronomy.org/meetings/wia2011/), which was held on 13 May 2011 in Sydney, Australia. The workshop was organised by the Astronomical Society of Australia's Chapter on Women in Astronomy, to discuss some of the issues that face women in astronomy and make recommendations to help support the success of women in Australian astronomy but came to broader conclusions that have value for the whole astronomical community. The workshop consisted of four sessions, with presentations by invited speakers on demographics, leadership, varie…
In this article we wonder what the next 100 years will bring for women in astronomy in the UK. After this year of looking back and celebrating 100 years of women in the Royal Astronomical Society (RAS), we now ask: what might the future hold? Extrapolating current trends, when might we expect equality in the genders of RAS members, speakers at meetings, award winners and more? Ultimately, when might we stop needing to talk about women in astronomy at all - when it will be as irrelevant to the conversation about astronomy as being a male astronomer is?
Measuring scientific development is a difficult task. Different metrics have been put forward to evaluate scientific development; in this paper we explore a metric that uses the number of peer-reviewed, and when available non-peer-reviewed articles, research research articles as an indicator of development in the field of astronomy. We analyzed the available publication record, using the SAO/NASA Astrophysics Database System, by country affiliation in the time span between 1950 and 2011 for countries with a Gross National Income of less than 14,365 USD in 2010. This represents 149 countries. W…
The American Astronomical Society's WorldWide Telescope (WWT) project enables terabytes of astronomical images, data, and stories to be viewed and shared among researchers, exhibited in science museums, projected into full-dome immersive planetariums and virtual reality headsets, and taught in classrooms from middle school to college levels. We review the WWT ecosystem, how WWT has been used in the astronomical community, and comment on future directions.
Several different controlled vocabularies have been developed and used by the astronomical community, each designed to serve a specific need and a specific group. The Unified Astronomy Thesaurus (UAT) attempts to provide a highly structured controlled vocabulary that will be relevant and useful across the entire discipline, regardless of content or platform. As two major use cases for the UAT include classifying articles and data, we examine the UAT in comparison with the Astronomical Subject Keywords used by major publications and the JWST Science Keywords used by STScI's Astronomer's Proposa…
In recent years, the Graphics Processing Unit (GPU) has emerged as a low-cost alternative for high performance computing, enabling impressive speed-ups for a range of scientific computing applications. Early adopters in astronomy are already benefiting in adapting their codes to take advantage of the GPU's massively parallel processing paradigm. I give an introduction to, and overview of, the use of GPUs in astronomy to date, highlighting the adoption and application trends from the first ~100 GPU-related publications in astronomy. I discuss the opportunities and challenges of utilising GPU co…
The orbital angular momentum (OAM) of light is potentially interesting for astronomical study of rotating objects such as black holes, but the effect of reduced spatial coherence of astronomical light sources such as stars is largely unknown. In a lab-scale experiment, we find that the detected OAM spectrum depends strongly on the position of the light-twisting object along the line of sight. We develop a simple intuitive model to predict the influence of reduced spatial coherence in astronomical observations, and discuss line-of-sight and intensity issues.
Data challenges are emerging as powerful tools with which to answer fundamental astronomical questions. Time-domain astronomy lends itself to data challenges, particularly in the era of classification and anomaly detection. With improved sensitivity of wide-field surveys in optical and radio wavelengths from surveys like the Large Synoptic Survey Telescope (LSST) and the Canadian Hydrogen Intensity Mapping Experiment (CHIME), we are entering the large-volume era of transient astronomy. I highlight some recent time-domain challenges, with particular focus on the Photometric LSST Astronomical Ti…
Data volumes from multiple sky surveys have grown from gigabytes into terabytes during the past decade, and will grow from terabytes into tens (or hundreds) of petabytes in the next decade. This exponential growth of new data both enables and challenges effective astronomical research, requiring new approaches. Thus far, astronomy has tended to address these challenges in an informal and ad hoc manner, with the necessary special expertise being assigned to e-Science or survey science. However, we see an even wider scope and therefore promote a broader vision of this data-driven revolution in a…
During the last 15 years the number of astronomy-related papers published by scientists in Venezuela has been continuously decreasing, mainly due to emigration. If rapid corrective actions are not implemented, Venezuelan astronomy could disappear.
In the distant past, astronomy was often intertwined with religion into a unified cosmos. As science became a distinct cultural enterprise, astronomy has witnessed a variety of rich interactions with other fields. Mathematical statistics was stimulated in the 19th century by astronomical problems, and today astrostatistics is a small but growing cross-disciplinary field advancing methodology to address challenges in astronomical data analysis. Throughout the 20th century, astronomy became closely allied with physics such that astronomy and astrophysics are now profoundly intertwined. Physical …
(Abridged) Canadian astronomy has, for decades, benefited from access to observatories and participating in international consortia on one of the best astronomical sites in the world: Maunakea. However, Maunakea is part of the unceded territory of the Native Hawaiian peoples and has always been of special significance to Hawaiian culture. The use of the summit and its science reserve has created tensions in the past decade, particularly with the development of the Thirty Meter Telescope. A meaningful and respectful response from the International astronomy community is still lacking. It is exp…
On the occasion of the International Year of Astronomy (IYA2009), we present a new interactive dictionary of astronomy and astrophysics, which contains about 7000 entries. This interdisciplinary and multicultural work is intended for professional and amateur astronomers, university students in astrophysics, as well as terminologists and linguists. A new approach is pursued in the formation of a scientific dictionary, which aims to display additional dimensions of astronomical concepts. Although Virtual Observatories recognize the necessity of efforts to define basic astronomical concepts and e…
Arts are a seamless way to introduce the general public to both basic and more sophisticated astronomical concepts. The visual richness of astronomy makes it attractive and easily incorporated in painting and literature. Astronomy is the only science with a muse - Urania - implying that, at least in the eyes of the ancients, it was an art itself. I review some less well known representation of astronomical concepts in literature with potential application in education.
Modern astronomy increasingly depends on computational thinking. Although some astronomy courses for undergraduates use computing, high school astronomy courses often have little computing. Created as a part of a research experience for teachers in astronomy and another in computer science, this project leverages robotic telescope images and astronomical algorithms to determine the distance to a star cluster using variable stellar photometry. Students investigate Python and Jupyter Notebook to analyze astronomical images to calculate the interstellar distance to a star cluster across the Milky…
The International Astronomical Youth Camp (IAYC) is a 50-year old summer camp, where participants work independently on astronomy projects. Due to the ongoing COVID-19 pandemic, the 2020 and 2021 instalments of the IAYC were cancelled, a first in the camp's history. An online format was established dubbed the eIAYC, consisting of three types of activities: (1) an engagement series with astronomical talks and workshops; (2) small independent research projects; and (3) a non-astronomical program involving a range of social activities. Here we present the experience of adapting an in-person camp …
The future of astronomy is inextricably entwined with the care and feeding of astronomical data products. Community standards such as FITS and NDF have been instrumental in the success of numerous astronomy projects. Their very success challenges us to entertain pragmatic strategies to adapt and evolve the standards to meet the aggressive data-handling requirements of facilities now being designed and built. We discuss characteristics that have made standards successful in the past, as well as desirable features for the future, and an open discussion follows.
The Virtual Observatory (VO) is realizing global electronic integration of astronomy data. One of the long-term goals of the U.S. VO project, the Virtual Astronomical Observatory (VAO), is development of services and protocols that respond to the growing size and complexity of astronomy data sets. This paper describes how VAO staff are active in such development efforts, especially in innovative strategies and techniques that recognize the limited operating budgets likely available to astronomers even as demand increases. The project has a program of professional outreach whereby new services …
The goal of this thesis is twofold; introduce the fundamentals of Bayesian inference and computation focusing on astronomical and cosmological applications, and present recent advances in probabilistic computational methods developed by the author that aim to facilitate Bayesian data analysis for the next generation of astronomical observations and theoretical models. The first part of this thesis familiarises the reader with the notion of probability and its relevance for science through the prism of Bayesian reasoning, by introducing the key constituents of the theory and discussing its best…
This review outlines concepts of mathematical statistics, elements of probability theory, hypothesis tests and point estimation for use in the analysis of modern astronomical data. Least squares, maximum likelihood, and Bayesian approaches to statistical inference are treated. Resampling methods, particularly the bootstrap, provide valuable procedures when distributions functions of statistics are not known. Several approaches to model selection and good- ness of fit are considered. Applied statistics relevant to astronomical research are briefly discussed: nonparametric methods for use when l…
Wavelets have been used extensively for several years now in astronomy for many purposes, ranging from data filtering and deconvolution, to star and galaxy detection or cosmic ray removal. More recent sparse representations such ridgelets or curvelets have also been proposed for the detection of anisotropic features such cosmic strings in the cosmic microwave background. We review in this paper a range of methods based on sparsity that have been proposed for astronomical data analysis. We also discuss what is the impact of Compressed Sensing, the new sampling theory, in astronomy for collectin…
The simulation of the optical turbulence (OT) for astronomical applications obtained with non-hydrostatic atmospherical models at meso-scale presents, with respect to measurements, some advantages. The future of the ground-based astronomy relies upon the potentialities and feasibility of the ELTs. Our ability in knowing, controlling and 'managing' the effects of the turbulence on such a new generation telescopes and facilities are determinant to assure their competitiveness with respect to the space astronomy. In the past several studies have been carried out proving the feasibility of the sim…
The infrared astronomy group of Department of Astronomy and Astrophysics at Tata Institute of Fundamental Research has been pursuing astronomical instrumentation activities since its inception. The group has been routinely involved in balloon-borne astronomy programs from the field station at Hyderabad with indigenously developed payloads. Ground-based astronomical activities began with a single element infrared detector. Later, over time, larger format array detectors are being used in the cameras. These astronomy cameras have been routinely used at observatories across India. Recently, the g…
The traditional cultures of Aboriginal Australians include a significant astronomical component, perpetuated through oral tradition, ceremony, and art. This astronomical component includes a deep understanding of the motion of objects in the sky, and this knowledge was used for practical purposes, such as constructing calendars. There is also evidence that traditional Aboriginal Australians made careful records and measurements of cyclical phenomena, paid careful attention to unexpected phenomena such as eclipses and meteorite impacts, and could determine the cardinal points to an accuracy of …
CASA, the Common Astronomy Software Applications, is the primary data processing software for the Atacama Large Millimeter/submillimeter Array (ALMA) and the Karl G. Jansky Very Large Array (VLA), and is frequently used also for other radio telescopes. The CASA software can handle data from single-dish, aperture-synthesis, and Very Long Baseline Interferometery (VLBI) telescopes. One of its core functionalities is to support the calibration and imaging pipelines for ALMA, VLA, VLA Sky Survey (VLASS), and the Nobeyama 45m telescope. This paper presents a high-level overview of the basic structu…
In a recent paper, published at arXiv:0910.2381, we started a discussion on the new possibilities arising from the use of fractional differential calculus in image processing. We have seen that the fractional calculation is able to enhance the quality of images, with interesting possibilities in edge detection and image restoration. Here, we want to discuss more deeply its role as a tool for the processing of astronomical images. In particular, the fractional differentiation can help produce a 'content-matter' based image from a pretty astronomical image that can be used for more research and …
From Timbuktu to SKA: Who owns the Astronomy knowledge Africa produces?
North-West University（西北大学）；Centre for Higher Education Professional Development (CHEPD)（高等教育专业发展中心）；The Education and Human Rights in Diversity Research Unit (Edu-HRight)（教育与人权多样性研究单元）；Centre for Space Research（空间研究中心）；National Institute for Theoretical and Computational Sciences (NITheCS)（理论与计算科学国家研究所）
With the implementation of many ambitious observation projects, including LAMOST, FAST, and Antarctic observatory at Doom A, observational astronomy in China is stepping into a brand new era with emerging data avalanche. In the era of e-Science, both these cutting-edge projects and traditional astronomy research need much more powerful data management, sharing and interoperability. Based on data-grid concept, taking advantages of the IVOA interoperability technologies, China-VO is developing a VO-driven astronomical data grid environment to enable multi-wavelength science and large database sc…
Allan variance (AVAR) was first introduced more than 40 years ago as a estimator of the stability of frequency standards, and now it is actively used for investigations of time series in astronomy, geodesy and geodynamics. This method allows one to effectively explore the noise characteristics for various data, such as variations of station and source coordinates, etc. Moreover, this technique can be used to investigate the spectral and fractal structure of the noise in measured data. To process unevenly weighted and multidimensional data, which are usual for many astronomy and geodesy applica…
Hierarchical clustering is a common algorithm in data analysis. It is unique among many clustering algorithms in that it draws dendrograms based on the distance of data under a certain metric, and group them. It is widely used in all areas of astronomical research, covering various scales from asteroids and molecular clouds, to galaxies and galaxy cluster. This paper systematically reviews the history and current status of the development of hierarchical clustering methods in various branches of astronomy. These applications can be grouped into two broad categories, one revealing the intrinsic…
Detection of a signal hidden by noise within a time series is an important problem in many astronomical searches, i.e. for light curves containing the contributions of periodic/semi-periodic components due to rotating objects and all other astrophysical time-dependent phenomena. One of the most popular tools for use in such studies is the "periodogram", whose use in an astronomical context is often not trivial. The "optimal" statistical properties of the periodogram are lost in the case of irregular sampling of signals, which is a common situation in astronomical experiments. Parts of these pr…
We occasionally include projects in our learner-centered introductory astronomy college course to enable non-science major students explore some astronomical concepts in more detail than otherwise. Such projects also highlight ongoing or upcoming astronomical events. We hope that students will feel more interested in astronomy through projects tied to astronomical events. In Spring 2012, we offered short essays focused on eclipses, transits and occultations to promote the rare transit of Venus that occurred on June 5th, 2012. We asked students to write two short essays from three that were off…
Where appropriate repositories are not available to support all relevant astronomical data products, data can fall into darkness: unseen and unavailable for future reference and re-use. Some data in this category are legacy or old data, but newer datasets are also often uncurated and could remain "dark". This paper provides a description of the design motivation and development of Astrolabe, a cyberinfrastructure project that addresses a set of community recommendations for locating and ensuring the long-term curation of dark or otherwise at-risk data and integrated computing. This paper also …
We report on the construction of a deep convolutional neural network that can reproduce the sensitivity of a matched-filtering search for binary black hole gravitational-wave signals. The standard method for the detection of well modeled transient gravitational-wave signals is matched filtering. However, the computational cost of such searches in low latency will grow dramatically as the low frequency sensitivity of gravitational-wave detectors improves. Convolutional neural networks provide a highly computationally efficient method for signal identification in which the majority of calculatio…
We derive the ranking of the astronomical observatories with the highest impact in astronomy based on the citation analysis of papers published in 2006. We also present a description of the methodology we use to derive this ranking. The current ranking is lead by the Sloan Digital Sky Survey, followed by Swift and the Hubble Space Telescope.
Astronomical observations already produce vast amounts of data through a new generation of telescopes that cannot be analyzed manually. Next-generation telescopes such as the Large Synoptic Survey Telescope and the Square Kilometer Array are planned to become operational in this decade and the next, and will increase the data volume by many orders of magnitude. The increased spatial, temporal and spectral resolution afford a powerful magnifying lens on the physical processes that underlie the data but, at the same time, generate unprecedented complexity hard to exploit for knowledge extraction…
Many estimation problems in astrophysics are highly complex, with high-dimensional, non-standard data objects (e.g., images, spectra, entire distributions, etc.) that are not amenable to formal statistical analysis. To utilize such data and make accurate inferences, it is crucial to transform the data into a simpler, reduced form. Spectral kernel methods are non-linear data transformation methods that efficiently reveal the underlying geometry of observable data. Here we focus on one particular technique: diffusion maps or more generally spectral connectivity analysis (SCA). We give examples o…
The traditional cultures of Aboriginal Australians include a significant astronomical component, perpetuated through oral tradition, ceremony, and art. This astronomical knowledge includes a deep understanding of the motion of objects in the sky, which was used for practical purposes such as constructing calendars and for navigation. There is also evidence that traditional Aboriginal Australians made careful records and measurements of cyclical phenomena, recorded unexpected phenomena such as eclipses and meteorite impacts, and could determine the cardinal points to an accuracy of a few degree…
Deep learning-based astronomical multimodal data fusion: A comprehensive review
National Astronomical Observatories, Chinese Academy of Sciences（中国科学院国家天文台）；University of Chinese Academy of Sciences（中国科学院大学）；National Astronomical Data Center（国家天文数据中心）
Kiwi Advanced Research and Education Network (KAREN) has been used to transfer large volumes of radio astronomical data between the AUT Radio Astronomical Observatory at Warkworth, New Zealand and the international organisations with which we are collaborating and conducting observations. Here we report on the current status of connectivity and on the results of testing different data transfer protocols. We investigate new UDP protocols such as "tsunami" and UDT and demonstrate that the UDT protocol is more efficient than "tsunami" and ftp. We report on our initial steps towards real-time eVLB…
AstroLLaVA: towards the unification of astronomical data and natural language
UniverseTBD；Indian Institute of Technology Delhi（印度理工学院德里分校）；Intelligent Internet Inc.（Intelligent Internet公司）；University of Florence（佛罗伦萨大学）；Instituto de Astrofísica de Canarias (IAC)（加那利群岛天体物理学研究所）；Observatoire de Paris（巴黎天文台）；PSL University（巴黎文理研究大学）；Université Paris-Cité（巴黎城市大学）；ANU RSAA（澳大利亚国立大学天体物理学与天体生物学研究学院）；European Space Agency（欧洲空间局）；IRISA（IRISA研究所）；Université Bretagne Sud（南布列塔尼大学）；ANU School of Computing（澳大利亚国立大学计算机学院）
Commodity cloud computing, as provided by commercial vendors such as Amazon, Google, and Microsoft, has revolutionized computing in many sectors. With the advent of a new class of big data, public access astronomical facility such as LSST, DKIST, and WFIRST, there exists a real opportunity to combine these missions with cloud computing platforms and fundamentally change the way astronomical data is collected, processed, archived, and curated. Making these changes in a cross-mission, coordinated way can provide unprecedented economies of scale in personnel, data collection and management, archi…
