Large language models (LLMs) have become the norm in natural language processing (NLP), excelling in few-shot in-context learning (ICL) with their remarkable abilities. Nonetheless, the success of ICL largely hinges on the choice of few-shot demonstration examples, making the selection process increasingly crucial. Existing methods have delved into optimizing the quantity and semantic similarity of these examples to improve ICL performances. However, our preliminary experiments indicate that the effectiveness of ICL is limited by the length of the input context. Moreover, varying combinations of few-shot demonstration examples can significantly boost accuracy across different test samples. To address this, we propose a novel method named parallel in-context learning (ParaICL) that effectively utilizes all demonstration examples without exceeding the manageable input context length. ParaICL employs parallel batching to distribute demonstration examples into different batches according to the semantic similarities of the questions in the demonstrations to the test question. It then computes normalized batch semantic scores for each batch. A weighted average semantic objective, constrained by adaptive plausibility, is applied to select the most appropriate tokens. Through extensive experiments, we validate the effectiveness of ParaICL and conduct ablation studies to underscore its design rationale. We further demonstrate that ParaICL can seamlessly integrate with existing methods.
ParallelComp: Parallel Long-Context Compressor for Length Extrapolation
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；The University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Sun Yat-Sen University（中山大学）；Tencent（腾讯）；Hong Kong Polytechnic University（香港理工大学）
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（工业和信息化部模式分析与机器智能重点实验室）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（教育部脑机智能技术重点实验室）
Although many context-aware neural machine translation models have been proposed to incorporate contexts in translation, most of those models are trained end-to-end on parallel documents aligned in sentence-level. Because only a few domains (and language pairs) have such document-level parallel data, we cannot perform accurate context-aware translation in most domains. We therefore present a simple method to turn a sentence-level translation model into a context-aware model by incorporating a document-level language model into the decoder. Our context-aware decoder is built upon only a sentenc…
Improving the Transformer Translation Model with Document-Level Context
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Sogou Inc.（搜狗公司）；Soochow University（苏州大学）
University of Washington（华盛顿大学）；Facebook AI（Facebook人工智能）
State-of-the-art neural machine translation models generate a translation from left to right and every step is conditioned on the previously generated tokens. The sequential nature of this generation process causes fundamental latency in inference since we cannot generate multiple tokens in each sentence in parallel. We propose an attention-masking based model, called Disentangled Context (DisCo) transformer, that simultaneously generates all tokens given different contexts. The DisCo transformer is trained to predict every output token given an arbitrary subset of the other reference tokens. …
Understanding In-Context Machine Translation for Low-Resource Languages: A Case Study on Manchu
Center for Information and Language Processing, LMU Munich（慕尼黑大学语言与信息处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Sorbonne Université, CNRS, ISIR（索邦大学，法国国家科学研究中心，智能系统与机器人研究所）
Self-Manager: Parallel Agent Loop for Long-form Deep Research
University of Chinese Academy of Sciences（中国科学院大学）；ModelBest Inc.（ModelBest公司）；University of Queensland（昆士兰大学）；University of California Merced（加州大学默塞德分校）
Conformer has proven to be effective in many speech processing tasks. It combines the benefits of extracting local dependencies using convolutions and global dependencies using self-attention. Inspired by this, we propose a more flexible, interpretable and customizable encoder alternative, Branchformer, with parallel branches for modeling various ranged dependencies in end-to-end speech processing. In each encoder layer, one branch employs self-attention or its variant to capture long-range dependencies, while the other branch utilizes an MLP module with convolutional gating (cgMLP) to extract…
Tongji University（同济大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper presents a novel open-domain dialogue generation model emphasizing the differentiation of speakers in multi-turn conversations. Differing from prior work that solely relies on the content of conversation history to generate a response, we argue that capturing relative social relations among utterances (i.e., generated by either the same speaker or different persons) benefits the machine capturing fine-grained context information from a conversation history to improve context coherence in the generated response. Given that, we propose a speaker-aware Parallel Hierarchical Attentive E…
Stop the Flip-Flop: Context-Preserving Verification for Fast Revocable Diffusion Decoding
King's College London, UK；Centre for AI, Data Science \& Artificial Intelligence, BioPharmaceuticals R\&D, AstraZeneca, UK；The Alan Turing Institute, UK；Imperial College London, UK
The two dominant approaches to neural text generation are fully autoregressive models, using serial beam search decoding, and non-autoregressive models, using parallel decoding with no output dependencies. This work proposes an autoregressive model with sub-linear parallel time generation. Noting that conditional random fields with bounded context can be decoded in parallel, we propose an efficient cascaded decoding approach for generating high-quality output. To parameterize this cascade, we introduce a Markov transformer, a variant of the popular fully autoregressive model that allows us to …
