To date, most abstractive summarisation models have relied on variants of the negative log-likelihood (NLL) as their training objective. In some cases, reinforcement learning has been added to train the models with an objective that is closer to their evaluation measures (e.g. ROUGE). However, the reward function to be used within the reinforcement learning approach can play a key role for performance and is still partially unexplored. For this reason, in this paper, we propose two reward functions for the task of abstractive summarisation: the first function, referred to as RwB-Hinge, dynamically selects the samples for the gradient update. The second function, nicknamed RISK, leverages a small pool of strong candidates to inform the reward. In the experiments, we probe the proposed approach by fine-tuning an NLL pre trained model over nine summarisation datasets of diverse size and nature. The experimental results show a consistent improvement over the negative log-likelihood baselines.
The University of Melbourne（墨尔本大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；RMIT University（皇家墨尔本理工大学）
In this paper we report on our submission to the Multidocument Summarisation for Literature Review (MSLR) shared task. Specifically, we adapt PRIMERA (Xiao et al., 2022) to the biomedical domain by placing global attention on important biomedical entities in several ways. We analyse the outputs of the 23 resulting models, and report patterns in the results related to the presence of additional global attention, number of training steps, and the input configuration.
Royal Holloway, University of London（伦敦大学皇家霍洛威学院）；Technische Universität Darmstadt（达姆施塔特工业大学）
Document summarisation can be formulated as a sequential decision-making problem, which can be solved by Reinforcement Learning (RL) algorithms. The predominant RL paradigm for summarisation learns a cross-input policy, which requires considerable time, data and parameter tuning due to the huge search spaces and the delayed rewards. Learning input-specific RL policies is a more efficient alternative but so far depends on handcrafted rewards, which are difficult to design and yield poor performance. We propose RELIS, a novel RL paradigm that learns a reward function with Learning-to-Rank (L2R) …
Technische Universität Darmstadt（达姆施塔特工业大学）；Bar-Ilan University（巴伊兰大学）
Reinforcement Learning (RL) based document summarisation systems yield state-of-the-art performance in terms of ROUGE scores, because they directly use ROUGE as the rewards during training. However, summaries with high ROUGE scores often receive low human judgement. To find a better reward function that can guide RL to generate human-appealing summaries, we learn a reward function from human ratings on 2,500 summaries. Our reward function only takes the document and system summary as input. Hence, once trained, it can be used to train RL-based summarisation systems without using any reference …
Imperial College London（帝国理工学院）；University of Sheffield（谢菲尔德大学）；Dublin City University（都柏林城市大学）
Reinforcement Learning (RL) is a powerful framework to address the discrepancy between loss functions used during training and the final evaluation metrics to be used at test time. When applied to neural Machine Translation (MT), it minimises the mismatch between the cross-entropy loss and non-differentiable evaluation metrics like BLEU. However, the suitability of these metrics as reward function at training time is questionable: they tend to be sparse and biased towards the specific words used in the reference texts. We propose to address this problem by making models less reliant on such me…
Exploring Generative Process Reward Modeling for Semi-Structured Data: A Case Study of Table Question Answering
University of Augsburg（奥格斯堡大学）；Institut für Maschinelle Sprachverarbeitung, University of Stuttgart（斯图加特大学机械语言处理研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究所）；NUS Graduate School for Integrative Sciences and Engineering（新加坡国立大学综合科学与工程研究生院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；ZhuiYi Technology（追一科技）；The MOE Key Laboratory of Computational Linguistics, Peking University（北京大学教育部计算语言学重点实验室）
Recent question generation (QG) approaches often utilize the sequence-to-sequence framework (Seq2Seq) to optimize the log-likelihood of ground-truth questions using teacher forcing. However, this training objective is inconsistent with actual question quality, which is often reflected by certain global properties such as whether the question can be answered by the document. As such, we directly optimize for QG-specific objectives via reinforcement learning to improve question quality. We design three different rewards that target to improve the fluency, relevance, and answerability of generate…
The Vanguard Group（先锋集团）；University of Toronto（多伦多大学）
Query reformulation aims to alter noisy or ambiguous text sequences into coherent ones closer to natural language questions. This is to prevent errors from propagating in a client-facing pipeline and promote better communication with users. Besides, it is crucial to maintain performance in downstream environments like question answering when rephrased queries are given as input. We show that under the previous framework (AQA), attempts to alter RL algorithms do not bring significant benefits to either reward acquisition or sequence fluency. Instead, we leverage a query-reformulating text-to-te…
MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Nanyang Technological University（南洋理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；MiroMind；The Chinese University of Hong Kong（香港中文大学）
Autonomous reinforcement learning agents, like children, do not have access to predefined goals and reward functions. They must discover potential goals, learn their own reward functions and engage in their own learning trajectory. Children, however, benefit from exposure to language, helping to organize and mediate their thought. We propose LE2 (Language Enhanced Exploration), a learning algorithm leveraging intrinsic motivations and natural language (NL) interactions with a descriptive social partner (SP). Using NL descriptions from the SP, it can learn an NL-conditioned reward function to f…
