Existing Large Language Models (LLMs) generate text through unidirectional autoregressive decoding methods to respond to various user queries. These methods tend to consider token selection in a simple sequential manner, making it easy to fall into suboptimal options when encountering uncertain tokens, referred to as chaotic points in our work. Many chaotic points exist in texts generated by LLMs, and they often significantly affect the quality of subsequently generated tokens, which can interfere with LLMs' generation. This paper proposes Self-Evaluation Decoding, SED, a decoding method for enhancing model generation. Analogous to the human decision-making process, SED integrates speculation and evaluation steps into the decoding process, allowing LLMs to make more careful decisions and thus optimize token selection at chaotic points. Experimental results across various tasks using different LLMs demonstrate SED's effectiveness.
Unifying Speech Editing Detection and Content Localization via Prior-Enhanced Audio LLMs
Key Laboratory of Aerospace Information Security(航空信息安全与可信计算重点实验室);School of Cyber Science and Engineering(网络安全工程学院);Wuhan University(武汉大学);Independent Researcher(独立研究员);School of Computer Science and Technology(计算机科学与技术学院);Anhui University(安徽大学);Communication University of China(中国通信大学);Beihang University(北京航空航天大学)
Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR
Alibaba-NTU Global e-Sustainability CorpLab(阿里巴巴-国立科技大学全球可持续发展公司实验室);Nanyang Technological University(国立科技大学);College of Computing and Data Science(计算与数据科学学院)
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Microsoft Dynamics 365 AI(微软Dynamics 365人工智能);Microsoft Research(微软研究院)
预训练神经语言模型的最新进展显着提高了许多自然语言处理(NLP)任务的性能。在本文中,我们提出了一个新的模型架构DeBERTa(解码增强BERT与解开注意),提高了BERT和RoBERTa模型使用两种新的技术。第一种是解纠缠的注意力机制,其中每个词使用两个向量来表示,这两个向量分别编码其内容和位置,并且词之间的注意力权重分别使用解纠缠矩阵来计算它们的内容和相对位置。第二,增强的掩码解码器用于在解码层中合并绝对位置,以在模型预训练中预测掩码令牌。此外,一种新的虚拟对抗训练方法用于微调,以提高模型的泛化能力。我们表明,这些技术显着提高模型预训练的效率和性能的自然语言理解(NLU)和自然语言生成(NLG)下游任务。与RoBERTa-Large相比,在一半训练数据上训练的DeBERTa模型在广泛的NLP任务上表现更好,MNLI提高了+0.9%(90.2% vs. 91.1%),SQuAD v2.0提高了+2.3%(88.4% vs. 90.7%),RACE提高了+3.6%(83.2% vs. 86.8%)。值得注意的是,我们通过训练一个由48个变换层和15亿个参数组成的更大版本来扩展DeBERTa。显著的性能提升使得单个DeBERTa模型在SuperGLUE基准上超过人类性能(Wang等人,2019年a)首次在宏观平均得分方面(89.9对89.8),并且截至2021年1月6日,整体DeBE…