发表机构
IIIT Kalyani; BITS Pilani Goa; IISER Kolkata（卡里亚尼印度信息技术学院; 皮拉尼贝拉理工学院果阿校区; 加尔各答印度科学教育与研究学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究探究能否通过将感知碳排放的联合损失机制用于微调，在Gemmc-2 2B等三类LLM上实现推理性能与碳排放的盈亏平衡，发现碳排放项作用随任务结构变化，提出轻量正则化方案。
AI 中文摘要
任何已部署的大语言模型（LLM）的碳足迹会在推理过程中累积，重复使用该模型产生的成本远超一次微调的成本。然而，大多数效率优化措施要么针对预训练规模，要么针对事后压缩。我们探究是否可以将经过校准的可微分能量替代项纳入微调目标，从而使推理行为在零或接近零的碳成本下获得任务准确率，即形成盈亏平衡配置。我们提出一种联合损失机制，该机制包含针对每个模型的碳排放参数、基于参数范数的线性替代项、FLOP代理项以及内存代理项，这些项通过硬件上的能量分析拟合得到。我们对三个架构不同的模型系列进行了微调：Gemma-2 2B、Llama-3.1 8B和Qwen-2.5 14B，并在MMLU的三个科目（抽象代数、哲学和形式逻辑）上评估推理的F1值和CO₂排放量。我们从多个结果中发现，碳排放项的作用取决于任务结构，要么是有害干扰，要么是有益正则化。我们将经过校准的感知碳排放微调定位为一种轻量、可直接使用的正则化器，具有非空但依赖于模型和任务的盈亏平衡区域。这是一项正在进行的工作，我们将很快发布代码库。
英文摘要
The carbon footprint of any deployed Large Language Model (LLM) accumulates during inference, where repeated use of the model substantially exceeds the one-time cost of fine-tuning. Yet most efficiency interventions target either pre-training scale or post-hoc compression. We ask whether folding a calibrated, differentiable energy surrogate into the fine-tuning objective can produce inference behavior that gains task accuracy at zero or near-zero carbon cost, a break-even configuration. We propose a joint loss mechanism with a per-model carbon-emission parameter, a linear surrogate over parameter norm, FLOP proxy, and a memory proxy, fit from on-hardware energy profiling. We fine-tune three architecturally distinct families: Gemma-2 2B, Llama-3.1 8B, and Qwen-2.5 14B, and evaluate inference F1 and CO$_2$ emissions on three MMLU subjects: abstract algebra, philosophy, and formal logic. We discover from several outcomes that the carbon term behaves as either harmful interference or beneficial regularization depending on the task structure. We position calibrated carbon-aware fine-tuning as a lightweight, drop-in regularizer with a non-empty but model and task-dependent break-even region. This is an ongoing work, and we will release our codebase soon.
Comments13 Pages, 6 Figures, Submitted to ARR Cycle
