We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures. Transformers faces severe computational and memory bottlenecks with long sequences due to the quadratic complexity of softmax attention and the growing Key-Value (KV) cache that makes inference memory-bound by context length. Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving model quality. Unlike prior linearization methods constrained by fixed, non-adaptive structures, Lizard augments the architecture with compact, learnable modules that enable adaptive memory control and robust length generalization. Moreover, we introduce a hardwareaware algorithm that solves numerical instability in gated attention to accelerate training. Extensive experiments show that Lizard achieves near-lossless recovery of its teacher model's performance, significantly outperforming previous methods by up to 9.4 - 24.5 points on the 5-shot MMLU benchmark and demonstrating superior associative recall.
Large Language Models Meet Knowledge Graphs to Answer Factoid Questions
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);Universität Hamburg(汉堡大学);Nazarbayev University(纳扎尔巴耶夫大学);ISP RAS(俄罗斯科学院信息安全问题研究所);Artificial Intelligence Research Institute(人工智能研究院)
Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室);Huawei Cloud(华为云);School of Information, Renmin University of China(中国人民大学信息学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
Orthogonal Hierarchical Decomposition for Structure-Aware Table Understanding with Large Language Models
Zhejiang University of Technology, China(浙江工业大学,中国);Zhejiang Key Laboratory of Visual Information Intelligent Processing, China(浙江视觉信息智能处理重点实验室,中国);University of Aberdeen, UK(爱丁堡大学,英国)
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
SMULL Group, Harbin Institute of Technology, Shenzhen(SMULL小组,哈尔滨工业大学,深圳);Pengcheng Laboratory(Pengcheng实验室);UQMM Lab, University of Queensland(UQMM实验室,昆士兰大学);Huawei Technologies Co., Ltd.(华为技术有限公司)