首页/论文/2002.09646

arXiv 2002.096462020-02cs.CL

Machine Translation System Selection from Bandit Feedback

Preferred Networks；Johns Hopkins University（约翰斯·霍普金斯大学）

Jason Naradowsky, Xuan Zhang, Kevin Duh

备注：: Accepted to AMTA 2020

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Adapting machine translation systems in the real world is a difficult problem. In contrast to offline training, users cannot provide the type of fine-grained feedback (such as correct translations) typically used for improving the system. Moreover, different users have different translation needs, and even a single user's needs may change over time. In this work we take a different approach, treating the problem of adaptation as one of selection. Instead of adapting a single system, we train many translation systems using different architectures, datasets, and optimization methods. Using bandit learning techniques on simulated user feedback, we learn a policy to choose which system to use for a particular translation task. We show that our approach can (1) quickly adapt to address domain changes in translation tasks, (2) outperform the single best system in mixed-domain translation tasks, and (3) make effective instance-specific decisions when using contextual bandit strategies.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Bandit Structured Prediction for Learning from Partial Feedback in Statistical Machine Translation

Heidelberg University（海德堡大学）；Orange Labs（Orange实验室）

We present an approach to structured prediction from bandit feedback, called Bandit Structured Prediction, where only the value of a task loss function at a single predicted point, instead of a correct structure, is observed in learning. We present an application to discriminative reranking in Statistical Machine Translation (SMT) where the learning algorithm only has access to a 1-BLEU loss evaluation of a predicted translation instead of obtaining a gold standard reference translation. In our experiment bandit feedback is obtained by evaluating BLEU on reference translations without revealin…

2016-01-18cs.CLcs.LG

Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits

Google Research（谷歌研究院）

Training data for machine translation (MT) is often sourced from a multitude of large corpora that are multi-faceted in nature, e.g. containing contents from multiple domains or different levels of quality or complexity. Naturally, these facets do not occur with equal frequency, nor are they equally important for the test scenario at hand. In this work, we propose to optimize this balance jointly with MT model parameters to relieve system developers from manual schedule design. A multi-armed bandit is trained to dynamically choose between facets in a way that is most beneficial for the MT syst…

2021-10-13cs.AIcs.CL

The UMD Neural Machine Translation Systems at WMT17 Bandit Learning Task

University of Maryland, College Park（马里兰大学帕克分校）

We describe the University of Maryland machine translation systems submitted to the WMT17 German-English Bandit Learning Task. The task is to adapt a translation system to a new domain, using only bandit feedback: the system receives a German sentence to translate, produces an English sentence, and only gets a scalar score as feedback. Targeting these two challenges (adaptation and bandit learning), we built a standard neural machine translation system and extended it in two ways: (1) robust reinforcement learning techniques to learn effectively from the bandit feedback, and (2) domain adaptat…

2017-08-07cs.AIcs.CLcs.HC

A Shared Task on Bandit Learning for Machine Translation

Amazon Development Center Germany（亚马逊德国开发中心）；Heidelberg University（海德堡大学）

We introduce and describe the results of a novel shared task on bandit learning for machine translation. The task was organized jointly by Amazon and Heidelberg University for the first time at the Second Conference on Machine Translation (WMT 2017). The goal of the task is to encourage research on learning machine translation from weak user feedback instead of human references or post-edits. On each of a sequence of rounds, a machine translation system is required to propose a translation for an input, and receives a real-valued estimate of the quality of the proposed translation for learning…

2017-07-27cs.CLstat.ML

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning

Ateneo de Manila University（马尼拉雅典耀大学）；Heidelberg University（海德堡大学）

We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task. Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra- and inter-annotator $α$-agreement is comparable. Best reliability is obtained for standardized…

2018-12-13cs.CLstat.ML

Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translation

Heidelberg University（海德堡大学）；Amazon Development Center（亚马逊开发中心）

The goal of counterfactual learning for statistical machine translation (SMT) is to optimize a target SMT system from logged data that consist of user feedback to translations that were predicted by another, historic SMT system. A challenge arises by the fact that risk-averse commercial SMT systems deterministically log the most probable translation. The lack of sufficient exploration of the SMT output space seemingly contradicts the theoretical requirements for counterfactual learning. We show that counterfactual learning from deterministic bandit logs is possible nevertheless by smoothing ou…

2017-12-14cs.CLcs.LGstat.ML

更多相关论文收起更多论文

Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback

University of Maryland（马里兰大学）；Microsoft Research, New York（微软研究院纽约分院）

Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training focuses on expensive human-generated reference translations. We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder-decoder architecture (Luong et al., 2015). This algor…

2017-11-11cs.AIcs.CLcs.HC

Bandit Structured Prediction for Neural Sequence-to-Sequence Learning

Heidelberg University（海德堡大学）

Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear bandit learning to neural sequence-to-sequence learning problems using attention-based recurrent neural networks. Furthermore, we show how to incorporate control variates into our learning algorithms for variance reduction and improved generalization. We present an evaluation on a …

2018-12-13cs.CLcs.LGstat.ML

Simulating Bandit Learning from User Feedback for Extractive Question Answering

Cornell University（康奈尔大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）

We study learning from user feedback for extractive question answering by simulating feedback using supervised data. We cast the problem as contextual bandit learning, and analyze the characteristics of several learning scenarios with focus on reducing data annotation. We show that systems initially trained on a small number of examples can dramatically improve given feedback from users on model-predicted answers, and that one can use existing datasets to deploy systems in new domains without any annotation, but instead improving the system on-the-fly via user feedback.

2022-03-18cs.CL

利用大语言模型与多臂老虎机实现非洲语言的领域内翻译

In-Domain African Languages Translation Using LLMs and Multi-armed Bandits

Media Analysis Group, Sony Research India（索尼印度研究机构媒体分析组）

针对低资源非洲语言领域内翻译中微调不可行、模型选择难的问题，提出基于多臂老虎机系列算法的NMT模型选择策略，在三种非洲语言的两类场景下验证了方法的鲁棒性与有效性。

2025-05-21cs.CL

Multi-Action Dialog Policy Learning from Logged User Feedback

JIUTIAN Team, China Mobile Research（中国移动研究院）

Multi-action dialog policy, which generates multiple atomic dialog actions per turn, has been widely applied in task-oriented dialog systems to provide expressive and efficient system responses. Existing policy models usually imitate action combinations from the labeled multi-action dialog examples. Due to data limitations, they generalize poorly toward unseen dialog flows. While reinforcement learning-based methods are proposed to incorporate the service ratings from real users and user simulators as external supervision signals, they suffer from sparse and less credible dialog-level rewards.…

2023-02-27cs.CL

Stochastic Structured Prediction under Bandit Feedback

Heidelberg University（海德堡大学）；Tufts University（塔夫茨大学）；Amazon Development Center（亚马逊开发中心）

Stochastic structured prediction under bandit feedback follows a learning protocol where on each of a sequence of iterations, the learner receives an input, predicts an output structure, and receives partial feedback in form of a task loss evaluation of the predicted structure. We present applications of this learning scenario to convex and non-convex objectives for structured prediction and analyze them as stochastic first-order methods. We present an experimental evaluation on problems of natural language processing over exponential output spaces, and compare convergence speed across differe…

2016-11-02cs.CLcs.LGstat.ML

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback

Heidelberg University（海德堡大学）

Counterfactual learning from human bandit feedback describes a scenario where user feedback on the quality of outputs of a historic system is logged and used to improve a target system. We show how to apply this learning framework to neural semantic parsing. From a machine learning perspective, the key challenge lies in a proper reweighting of the estimator so as to avoid known degeneracies in counterfactual learning, while still being applicable to stochastic gradient optimization. To conduct experiments with human users, we devise an easy-to-use interface to collect human feedback on semanti…

2018-11-30cs.CLcs.LGstat.ML