AstroPT:面向天文学的大规模观测模型
AstroPT: Scaling Large Observation Models for Astronomy
AI总结:
本文提出AstroPT,一种基于860万张星系图像预训练的自回归Transformer,发现其遵循对数缩放定律,并开源代码、权重与数据以推动社区协作。
摘要:
本文介绍了AstroPT,一种为天文用例而开发的自回归预训练Transformer。本文所展示的AstroPT模型已在来自DESI Legacy Survey DR8的860万张$512 \ imes 512$像素的$grz$波段星系邮票图像上进行了预训练。我们训练了一系列规模从100万到21亿参数不断增大的基础模型,并发现AstroPT遵循与文本模型相似的饱和对数-对数缩放定律。我们还发现,通过线性探测衡量的模型在下游任务上的性能随模型规模增大而提升,直至达到模型参数饱和点。我们认为,社区协作开发是实现开源“大规模观测模型”的最佳途径——即一种在观测科学数据上以自然语言处理规模进行训练的模型。为此,我们在MIT许可证下发布了AstroPT的源代码、权重和数据集,并邀请潜在合作者加入我们,共同构建和研究这些模型。
英文摘要:
This work presents AstroPT, an autoregressive pretrained transformer developed with astronomical use-cases in mind. The AstroPT models presented here have been pretrained on 8.6 million $512 \times 512$ pixel $grz$-band galaxy postage stamp observations from the DESI Legacy Survey DR8. We train a selection of foundation models of increasing size from 1 million to 2.1 billion parameters, and find that AstroPT follows a similar saturating log-log scaling law to textual models. We also find that the models' performances on downstream tasks as measured by linear probing improves with model size up to the model parameter saturation point. We believe that collaborative community development paves the best route towards realising an open source `Large Observation Model' -- a model trained on data taken from the observational sciences at the scale seen in natural language processing. To this end, we release the source code, weights, and dataset for AstroPT under the MIT license, and invite potential collaborators to join us in collectively building and researching these models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。