2024/05/23 by Michael J. Smith, Ryan J. Roberts, Smith, Michael J. +5 · 1 voice · 8 citations
Computer Science · Engineering · Mathematics · Physics and Astronomy · #Adaptive optics and wavefront sensing #Astronomical Observations and Instrumentation #Astronomy #Astronomy and Astrophysical Research #Astrophysics #Astrophysics of Galaxies (astro-ph.GA) #FOS: Computer and information sciences #FOS: Physical sciences #Geometry #Instrumentation and Methods for Astrophysics (astro-ph.IM) #Machine Learning (cs.LG) #Mathematics #Physics #Scaling #astro-ph.GA #astro-ph.IM #cs.LG
paper · pdf · doi:10.48550/arxiv.2405.14930
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/05/23 · arxiv published 2024/05/23 · arxiv updated 2024/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This work presents AstroPT, an autoregressive pretrained transformer developed with astronomical use-cases in mind. The AstroPT models presented here have been pretrained on 8.6 million 512 × 512 pixel grz-band galaxy postage stamp observations from the DESI Legacy Survey DR8. We train a selection of foundation models of increasing size from 1 million to 2.1 billion parameters, and find that AstroPT follows a similar saturating log-log scaling law to textual models. We also find that the models' performances on downstream tasks as measured by linear probing improves with model size up to the model parameter saturation point. We believe that collaborative community development paves the best route towards realising an open source `Large Observation Model' -- a model trained on data taken from the observational sciences at the scale seen in natural language processing. To this end, we release the source code, weights, and dataset for AstroPT under the MIT license, and invite potential collaborators to join us in collectively building and researching these models.