arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.23665cs.LGcs.AI
收藏

基于压缩表示的Transformer

Transformers from Compressed Representations

King Abdullah University of Science and Technology(卡塔尔国王 Abdullah 科学与技术大学);Meta AI Research(Meta AI 研究)

Juan C. Leon Alcazar, Mattia Soldan, Mohammad Saatialsoruji, Alejandro Pardo, Hani Itani, Juan Camilo Perez, Bernard Ghanem

AI总结:

针对压缩表示在表示学习中潜力未被充分挖掘的问题,提出TEMPEST方法,利用压缩文件字节流结构设计分词编码策略,让标准Transformer直接从压缩数据学习语义表示,在保证准确率的同时提升内存与计算效率。

摘要:

压缩文件格式是高效数据存储与传输的基石,但其在表示学习中的潜力仍未得到充分探索。我们提出TEMPEST(基于压缩表示的Transformer)方法,该方法利用压缩文件固有的字节流结构,设计出一种有效的分词与编码策略。借助这种紧凑编码,标准Transformer可直接从压缩数据流中学习语义表示,无需进行原始字节级处理或完整媒体解码。我们的方案大幅减少了语义分类所需的词元数量,从而降低了计算复杂度与内存占用。通过在不同数据集、编码方案及模态上开展的大量实验,我们表明TEMPEST在准确率上与当前最优水平相当,同时在内存与计算方面实现了效率提升。

英文摘要:

Compressed file formats are the corner stone of efficient data storage and transmission, yet their potential for representation learning remains largely underexplored. We introduce TEMPEST (TransformErs froM comPressed rEpreSenTations), a method that exploits the inherent byte-stream structure of compressed files to design an effective tokenization and encoding strategy. By leveraging this compact encoding, a standard transformer can directly learn semantic representations from compressed data streams, bypassing the need for raw byte-level processing or full media decoding. Our proposal substantially reduces the number of tokens required for semantic classification, thereby lowering both computational complexity and memory usage. Through extensive experiments across diverse datasets, coding schemes, and modalities, we show that TEMPEST achieves accuracy competitive wit the state-of-the-art while delivering efficiency gains in memory and compute.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑