A popular strategy to train recurrent neural networks (RNNs), known as ``teacher forcing'' takes the ground truth as input at each time step and makes the later predictions partly conditioned on those inputs. Such training strategy impairs their ability to learn rich distributions over entire sequences because the chosen inputs hinders the gradients back-propagating to all previous states in an end-to-end manner. We propose a fully differentiable training algorithm for RNNs to better capture long-term dependencies by recovering the probability of the whole sequence. The key idea is that at each time step, the network takes as input a ``bundle'' of similar words predicted at the previous step instead of a single ground truth. The representations of these similar words forms a convex hull, which can be taken as a kind of regularization to the input. Smoothing the inputs by this way makes the whole process trainable and differentiable. This design makes it possible for the model to explore more feasible combinations (possibly unseen sequences), and can be interpreted as a computationally efficient approximation to the beam search. Experiments on multiple sequence generation tasks yield performance improvements, especially in sequence-level metrics, such as BLUE or ROUGE-2.
Event Nugget Detection with Forward-Backward Recurrent Neural Networks
Oregon State University(俄勒冈州立大学)
传统的事件检测方法严重依赖于手动设计的丰富功能。最近的深度学习方法通过自动特征工程来缓解这个问题。但是,这些努力,像传统的方法,到目前为止只集中在单一的标记事件提到,而在实践中,事件也可以是一个短语。相反,我们使用前向-后向循环神经网络(FBRNN)来检测可能是单词或短语的事件。据我们所知,这是处理多词事件的首批努力之一,也是首次尝试使用RNN进行事件检测。实验结果表明,FBRNN在ACE 2005和Rich ERE 2015事件检测任务上与最先进的方法具有竞争力。
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University(吉林大学符号计算与知识工程重点实验室);Tongyi Lab, Alibaba Group(阿里集团通义实验室);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院)
Augmenting Neural Machine Translation with Knowledge Graphs
University of Paderborn(帕德博恩大学);Insight Centre for Data Analytics, Data Science Institute National University of Ireland Galway(爱尔兰国立高威大学数据科学研究院洞察数据分析中心)