趋同的世界表征与分歧的任务
Convergent World Representations and Divergent Tasks
AI总结：
本研究通过多任务训练产生趋同的世界表征，但存在分歧任务会损害新实体的整合与泛化。
摘要：
尽管神经表征是现代深度学习的核心，但支配其几何特性和其在下游适应性中的作用的条件仍理解甚微。我们开发了一个框架，明确区分底层世界、数据生成过程和由此产生的模型表征，以在受控环境中研究这些问题。5075个城市坐标定义了世界，7个几何任务生成自回归训练的数据。我们发现，不同任务导致了定性和定量上不同的世界表征几何。然而，多任务训练推动了世界表征的收敛：在非重叠任务上训练的模型发展出对齐的几何表征，提供了对柏拉图表征假说的多任务扩展假说的受控证据。为了研究适应性，我们先在所有任务上预训练模型，然后测试新实体（城市）是否能通过微调一致地整合到表征空间中。令人惊讶的是，我们发现尽管进行了多任务预训练，某些任务，我们称之为分歧的，会主动损害新实体的表征整合，并损害泛化能力。我们的结果表明，训练多个关系任务可靠地产生趋同的世界表征，但隐藏的分歧任务可通过微调严重损害新实体的整合。
英文摘要：
While neural representations are central to modern deep learning, the conditions governing their geometry and their roles in downstream adaptability remain poorly understood. We develop a framework clearly separating the underlying world, the data generation process and the resulting model representations to study these questions in a controlled setup. 5,075 city coordinates define the world and 7 geometric tasks generate the training data for autoregressive training. We find that different tasks give rise to qualitatively and quantitatively distinct world representation geometries. However, multi-task training drives convergence of world representations: models trained on non-overlapping tasks develop aligned geometric representations, providing controlled evidence for the Multitask Scaling Hypothesis of the Platonic Representation Hypothesis. To study adaptation, we pretrain models on all tasks, then test whether new entities (cities) can be consistently integrated into the representation space via fine-tuning. Surprisingly, we find that despite multi-task pretraining, some tasks, which we call divergent, actively harm the representational integration of new entities and harm generalization. Our results show that training on multiple relational tasks reliably produces convergent world representations, but lurking divergent tasks can catastrophically harm new entity integration via fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
