TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling

通过切片特征蒸馏实现神经网络的快速张量化

Fast Tensorization of Neural Networks via Slice-wise Feature Distillation

本文提出了一种基于切片特征蒸馏的可扩展张量化框架，用于神经网络压缩。该方法通过将网络分解为独立的切片（如单个层或块），并独立张量化每个切片以恢复原始预训练模型的中间表示，从而提高精度恢复、减少数据需求并实现高效的并行优化。

Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes

Quantization is widely employed in both cloud and edge systems to reduce the memory occupation, latency, and energy consumption of deep neural networks. In particular, mixed-precision quantization, i.e., the use of different bit-widths for different portions of the network, has been shown to provide excellent efficiency gains with limited accuracy drops, especially with optimized bit-width assignments determined by automated Neural Architecture Search (NAS) tools. State-of-the-art mixed-precision works layer-wise, i.e., it uses different bit-widths for the weights and activations tensors of ea…

One Size Does Not Fit All：一种面向更精确模型合并的分布感知稀疏化方法

One Size Does Not Fit All: Distribution-Guided Tensor-Wise Adaptive Sparsification for Model Merging

针对模型合并中统一稀疏率忽视参数异质性的问题，提出TADrop自适应稀疏化策略，根据张量分布特性分配定制稀疏级别，显著提升多种合并方法在多任务上的性能。

Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation

The past several years have witnessed Variational Auto-Encoder's superiority in various text generation tasks. However, due to the sequential nature of the text, auto-regressive decoders tend to ignore latent variables and then reduce to simple language models, known as the KL vanishing problem, which would further deteriorate when VAE is combined with Transformer-based structures. To ameliorate this problem, we propose DELLA, a novel variational Transformer framework. DELLA learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with th…

Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together

Neural networks equipped with self-attention have parallelizable computation, light-weight structure, and the ability to capture both long-range and local dependencies. Further, their expressive power and performance can be boosted by using a vector to measure pairwise dependency, but this requires to expand the alignment matrix to a tensor, which results in memory and computation bottlenecks. In this paper, we propose a novel attention mechanism called "Multi-mask Tensorized Self-Attention" (MTSA), which is as fast and as memory-efficient as a CNN, but significantly outperforms previous CNN-/…

张量与序列并行用于内存高效的Transformer训练与推断

Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference

本文提出TSP并行策略，通过在同一设备轴上融合张量并行与序列并行，减少内存开销，适用于长上下文和内存受限的模型训练，同时可与现有并行方案结合使用。