碳税变压器:用于过度语言模型的绿色压缩流程
Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models
AI总结:
本文提出碳税变压器,通过经济碳税原理设计压缩流程,提升软件工程中大语言模型的效率与环保性能,实现内存、时间及碳排放的显著降低,同时保持高精度。
摘要:
大型语言模型(LLMs)在软件工程中的快速采用带来了隐性的危机:不可持续的计算成本。尽管这些模型在不同SE任务中表现出色,但它们体积庞大、部署缓慢、内存消耗大且碳排放高。此研究挑战在于必须超越准确性,将效率和环境成本作为首要设计约束。为此,我们引入碳税变压器(CTT),一种受经济碳税原理启发的系统多架构压缩流程。基于碳定价概念,CTT实施计算碳税,惩罚低效架构并奖励可部署的压缩。我们评估了CTT在三个核心SE任务中的表现:代码克隆检测、代码摘要和代码生成,涵盖编码器-only、编码器-解码器和解码器-only架构。结果表明,CTT在推理方面实现了内存减少达49倍,克隆检测时间减少8-10倍,摘要时间减少3倍,生成时间减少4-7倍,碳排放减少81%,同时保持克隆检测约98%的精度,摘要约89%的精度,生成达91%(文本指标)和68%(pass@1)。两项消融研究显示,流程排序和单个组件贡献均至关重要,为CTT的设计和有效性提供了实证依据。本文为通过激进且性能保持的压缩实现负责任的AI在SE中的可行路径奠定了基础。
英文摘要:
The accelerating adoption of Large Language Models (LLMs) in software engineering (SE) has brought with it a silent crisis: unsustainable computational cost. While these models demonstrate remarkable capabilities in different SE tasks, they are unmanageably large, slow to deploy, memory-intensive, and carbon-heavy. This reality threatens not only the scalability and accessibility of AI-powered SE, but also its long-term environmental sustainability. The research challenge is clear: we must go beyond accuracy and address efficiency and environmental cost as first-class design constraints. To meet this challenge, we introduce Carbon-Taxed Transformers (CTT), a systematic multi-architectural compression principled pipeline ordering inspired by economic carbon taxation principles. Drawing from the economic concept of carbon pricing, CTT operationalizes a computational carbon tax that penalizes architectural inefficiencies and rewards deployment-ready compression. We evaluate CTT across three core SE tasks: code clone detection, code summarization, and code generation, with models spanning encoder-only, encoder-decoder, and decoder-only architecture. Our results show that CTT delivers on inference: (1) up to 49x memory reduction, (2) time reduction up to 8-10x for clone detection, up to 3x for summarization, and 4-7x for generation, (3) up to 81% reduction in CO2 emissions and (4) CTT retains around 98% accuracy on clone detection, around 89% on summarization, and up to 91% (textual metrics) and 68% (pass@1) for generation. Two ablation studies show that pipeline ordering and individual component contributions are both essential, providing empirical justification for CTT's design and effectiveness. This work establishes a viable path toward responsible AI in SE through aggressive yet performance-preserving compression.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。