Adjusting the latency, power, and accuracy of natural language understanding models is a desirable objective of an efficient architecture. This paper proposes an efficient Transformer architecture that adjusts the inference computational cost adaptively with a desired inference latency speedup. In fine-tuning phase, the proposed method detects less important hidden sequence elements (word-vectors) and eliminates them in each encoder layer using a proposed Attention Context Contribution (ACC) metric. After the fine-tuning phase, with the novel offline-tuning property, the inference latency of the model can be adjusted in a wide range of inference speedup selections without any further training. Extensive experiments reveal that most word-vectors in higher Transformer layers contribute less to subsequent layers, allowing their removal to improve inference latency. Experimental results on various language understanding, text generation, and instruction tuning tasks and benchmarks demonstrate the approach's effectiveness across diverse datasets, with minimal impact on the input's global context. The technique improves Time-to-First-Token (TTFT) of Llama3 by up to 2.9x, with minor performance drop. The suggested approach posits that in Large Language Models (LLMs), although the complete network is necessary for training, it can be truncated during the fine-tuning phase.
Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models
由于硬件和软件测试场景的多样性,在基于预训练语言模型(PLM)的IR模型评估时,往往忽略了延迟和效率问题。然而,效率是这种制度的一个重要部分,不应被忽视。在本文中,我们专注于提高效率的SPLADE模型,因为它已经取得了最先进的zero-shot性能和竞争力的结果TREC集合。SPLADE效率可以通过正则化因子来控制,但是单独控制该正则化已经被证明是不够有效的。为了减少SPLADE和传统检索系统之间的延迟差距,我们提出了几种技术,包括查询的L1正则化,文档/查询编码器的分离,FLOPS正则化的中间训练,以及使用更快的查询编码器。我们的基准测试表明,我们可以大大提高这些模型的效率,同时提高域内数据的性能指标。据我们所知,{我们提出了第一个神经模型,在相同的计算约束下,\textit{achieve similar latency (less than 4ms difference) as traditional BM25},同时具有\textit{similar performance (less than 10\% MRR@10 reduction)}作为最先进的单阶段神经排名器在域内数据}。
New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Models
Data Science(数据科学);Computer Philology and History of Contemporary German Literature(计算机语言学与当代德语文学史);CAIDAS -- Center for Artificial Intelligence and Data Science(CAIDAS—人工智能与数据科学中心)
Small Encoders Can Rival Large Decoders in Detecting Groundedness
Chandar Research Lab(昌达尔研究实验室);Mila – Quebec AI Institute(魁北克人工智能研究院);Université de Montréal(蒙特利尔大学);Polytechnique Montréal(蒙特利尔理工学院);Aily Labs(Aily实验室);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
WhisperRT -- Turning Whisper into a Causal Streaming Model
Andrew and Erna Viterbi Faculty of Electrical and Computer Engineering, Technion–Israel Institute of Technology(以色列理工学院安德鲁与厄娜·维特比电气与计算机工程学院);Language Technologies Institute, School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机科学学院语言技术研究所)