发表机构
University of Cambridge; NVIDIA; dClimate Labs（剑桥大学; 英伟达; dClimate实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究逐像素地球观测基础模型的扩展及预训练预算分配，通过大规模实验发现预训练损失难预测下游性能，给出计算分配规则，训练并蒸馏模型，其成果优于其他模型。
AI 中文摘要
逐像素地球观测（EO）基础模型通过生成空间嵌入实现了先进性能。但模型如何扩展以及如何最佳分配预训练预算仍不清楚。我们进行了迄今为止最大规模的EO控制缩放研究，发现预训练损失几乎无法预测下游性能，给出计算分配规则，训练并蒸馏模型，其成果优于其他模型，还将发布全球嵌入。
英文摘要
Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs within a fixed pixel-wise Barlow Twins family, each evaluated on 15 diverse downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise teachers (0.5B, 1B, and 2B) and distil the largest into compact students for embeddings-as-data deployment. In aggregate, our 44-million-parameter distilled student outperforms every open and proprietary embedding product we test, several of them an order of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. We plan to release global 10 m annual embeddings covering 2017-2025 as version 2 of the TESSERA foundation-model embeddings product. All code is available at: https://github.com/ucam-eo/tessera
