降温还是不降温？Temperature Network 通过 DRO 与大型基础模型相遇
To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO
AI总结：
本文提出 TempNet 温度预测网络，利用受约束 DRO 的鲁棒损失为 LFM 输入预测个性化温度，并在 LLM 与 CLIP 上验证其性能提升、泛化和任务迁移能力。
摘要：
温度参数在大型基础模型（LFM，例如大型语言模型（LLM）和 CLIP 模型）的训练和/或推理过程中发挥着深远作用。具体而言，它会调整 LLM 中 softmax 函数内的 logits，这对下一 token 生成至关重要；同时它会缩放训练 CLIP 模型时对比损失中的相似度。一个重要问题仍然存在：是否可行学习一个神经网络来预测任意输入数据的个性化温度，以增强 LFM？本文提出一个有原则的框架，用于学习一个小型但具有泛化能力的温度预测网络（TempNet），以改进 LFM。我们的解决方案由一个新颖的学习框架和一个经过恰当设计、具有理论启发的 TempNet 组成；该框架采用受约束分布鲁棒优化（DRO）支撑的鲁棒损失。TempNet 可以与大型基础模型一起从头训练，也可以在给定预训练基础模型的情况下单独学习。它不仅可用于预测个性化温度以促进 LFM 训练，而且能够泛化并迁移到新任务。我们在 LLM 和 CLIP 模型上的实验表明，TempNet 显著提升了现有解决方案或模型的性能，例如表 1。复现本文实验结果的代码可在 https://github.com/zhqiu/TempNet 获取。
英文摘要：
The temperature parameter plays a profound role during training and/or inference with large foundation models (LFMs) such as large language models (LLMs) and CLIP models. Particularly, it adjusts the logits in the softmax function in LLMs, which is crucial for next token generation, and it scales the similarities in the contrastive loss for training CLIP models. A significant question remains: Is it viable to learn a neural network to predict a personalized temperature of any input data for enhancing LFMs"? In this paper, we present a principled framework for learning a small yet generalizable temperature prediction network (TempNet) to improve LFMs. Our solution is composed of a novel learning framework with a robust loss underpinned by constrained distributionally robust optimization (DRO), and a properly designed TempNet with theoretical inspiration. TempNet can be trained together with a large foundation model from scratch or learned separately given a pretrained foundation model. It is not only useful for predicting personalized temperature to promote the training of LFMs but also generalizable and transferable to new tasks. Our experiments on LLMs and CLIP models demonstrate that TempNet greatly improves the performance of existing solutions or models, e.g. Table 1. The code to reproduce the experimental results in this paper can be found at https://github.com/zhqiu/TempNet.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
