首页/论文/2002.01535

arXiv 2002.015352020-02-04cs.CLcs.LG

Lightweight Convolutional Representations for On-Device Natural Language Processing

The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Facebook（脸书）

Shrey Desai, Geoffrey Goh, Arun Babu, Ahmed Aly

备注：: Accepted to MLSys 2020

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

The increasing computational and memory complexities of deep neural networks have made it difficult to deploy them on low-resource electronic devices (e.g., mobile phones, tablets, wearables). Practitioners have developed numerous model compression methods to address these concerns, but few have condensed input representations themselves. In this work, we propose a fast, accurate, and lightweight convolutional representation that can be swapped into any neural model and compressed significantly (up to 32x) with a negligible reduction in performance. In addition, we show gains over recurrent representations when considering resource-centric metrics (e.g., model file size, latency, memory usage) on a Samsung Galaxy S9.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Dilated Convolutional Neural Networks for Lightweight Diacritics Restoration

Eötvös Loránd University（罗兰大学）

Diacritics restoration has become a ubiquitous task in the Latin-alphabet-based English-dominated Internet language environment. In this paper, we describe a small footprint 1D dilated convolution-based approach which operates on a character-level. We find that solutions based on 1D dilated convolutional neural networks are competitive alternatives to models based on recursive neural networks or linguistic modeling for the task of diacritics restoration. Our solution surpasses the performance of similarly sized models and is also competitive with larger models. A special feature of our solutio…

2022-01-18cs.AIcs.CLcs.LG

DialogConv: A Lightweight Fully Convolutional Network for Multi-view Response Selection

Northeastern University（东北大学）；Singapore Management University（新加坡管理大学）

Current end-to-end retrieval-based dialogue systems are mainly based on Recurrent Neural Networks or Transformers with attention mechanisms. Although promising results have been achieved, these models often suffer from slow inference or huge number of parameters. In this paper, we propose a novel lightweight fully convolutional architecture, called DialogConv, for response selection. DialogConv is exclusively built on top of convolution to extract matching features of context and response. Dialogues are modeled in 3D views, where DialogConv performs convolution operations on embedding view, wo…

2022-10-25cs.CL

Lightweight Convolutional Approaches to Reading Comprehension on SQuAD

Stanford University（斯坦福大学）

Current state-of-the-art reading comprehension models rely heavily on recurrent neural networks. We explored an entirely different approach to question answering: a convolutional model. By their nature, these convolutional models are fast to train and capture local dependencies well, though they can struggle with longer-range dependencies and thus require augmentation to achieve comparable performance to RNN-based models. We conducted over two dozen controlled experiments with convolutional models and various kernel/attention/regularization schemes to determine the precise performance gains of…

2018-10-19cs.AIcs.CL

Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

University of Maryland（马里兰大学）；Nanyang Technological University（南洋理工大学）；Amazon AI（亚马逊人工智能）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Facebook AI（脸书人工智能）；UNSW（新南威尔士大学）；Singapore Management University（新加坡管理大学）；Mila；Polytechnique Montréal（蒙特利尔理工大学）

Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly ($75\%$) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Qua…

2019-06-11cs.CLcs.LG

Lightweight, Dynamic Graph Convolutional Networks for AMR-to-Text Generation

Singapore University of Technology and Design（新加坡科技设计大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Westlake University（西湖大学）；University of Edinburgh（爱丁堡大学）；ZJU-UIUC Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院）

AMR-to-text generation is used to transduce Abstract Meaning Representation structures (AMR) into text. A key challenge in this task is to efficiently learn effective graph representations. Previously, Graph Convolution Networks (GCNs) were used to encode input AMRs, however, vanilla GCNs are not able to capture non-local information and additionally, they follow a local (first-order) information aggregation scheme. To account for these issues, larger and deeper GCN models are required to capture more complex interactions. In this paper, we introduce a dynamic fusion mechanism, proposing Light…

2020-10-09cs.CL

Convolutional neural network compression for natural language processing

AGH University of Science and Technology（AGH科技大学）

Convolutional neural networks are modern models that are very efficient in many classification tasks. They were originally created for image processing purposes. Then some trials were performed to use them in different domains like natural language processing. The artificial intelligence systems (like humanoid robots) are very often based on embedded systems with constraints on memory, power consumption etc. Therefore convolutional neural network because of its memory capacity should be reduced to be mapped to given hardware. In this paper, results are presented of compressing the efficient co…

2018-05-28cs.CLcs.LGcs.NE

更多相关论文收起更多论文

Learning Context-Sensitive Convolutional Filters for Text Processing

Duke University（杜克大学）；NEC Laboratories America（美国NEC实验室）

Convolutional neural networks (CNNs) have recently emerged as a popular building block for natural language processing (NLP). Despite their success, most existing CNN models employed in NLP share the same learned (and static) set of filters for all input sentences. In this paper, we consider an approach of using a small meta network to learn context-sensitive convolutional filters for text processing. The role of meta network is to abstract the contextual information of a sentence or document into a set of input-aware filters. We further generalize this framework to model sentence pairs, where…

2018-08-30cs.CLcs.LGstat.ML

Very Deep Convolutional Networks for Text Classification

Facebook AI Research（Facebook人工智能研究院）；University of Le Mans（勒芒大学）

The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which have pushed the state-of-the-art in computer vision. We present a new architecture (VDCNN) for text processing which operates directly at the character level and uses only small convolutions and pooling operations. We are able to show that the performance of this model increases with depth: using up to 29 convolutional layers, we report improvements over the state-of…

2017-01-27cs.CLcs.LGcs.NE

Efficient Purely Convolutional Text Encoding

Institute of Computer Science（计算机科学研究所）；University of Wrocław（弗罗茨瓦夫大学）

In this work, we focus on a lightweight convolutional architecture that creates fixed-size vector embeddings of sentences. Such representations are useful for building NLP systems, including conversational agents. Our work derives from a recently proposed recursive convolutional architecture for auto-encoding text paragraphs at byte level. We propose alternations that significantly reduce training time, the number of parameters, and improve auto-encoding accuracy. Finally, we evaluate the representations created by our model on tasks from SentEval benchmark suite, and show that it can serve as…

2018-08-03cs.CL