Often when multiple labels are obtained for a training example it is assumed that there is an element of noise that must be accounted for. It has been shown that this disagreement can be considered signal instead of noise. In this work we investigate using soft labels for training data to improve generalization in machine learning models. However, using soft labels for training Deep Neural Networks (DNNs) is not practical due to the costs involved in obtaining multiple labels for large data sets. We propose soft label memorization-generalization (SLMG), a fine-tuning approach to using soft labels for training DNNs. We assume that differences in labels provided by human annotators represent ambiguity about the true label instead of noise. Experiments with SLMG demonstrate improved generalization performance on the Natural Language Inference (NLI) task. Our experiments show that by injecting a small percentage of soft label training data (0.03% of training set size) we can improve generalization performance over several baselines.
Reason to Rote: Rethinking Memorization in Reasoning
Department of ICS, Utrecht University(信息科学与计算系,乌特勒支大学);MaiNLP, Center for Information and Language Processing, LMU Munich(MaiNLP,信息与语言处理中心,慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Saarland Informatics Campus, Saarland University(萨尔兰州信息科学校区,萨尔兰州大学)
Unlocking Memorization in Large Language Models with Dynamic Soft Prompting
George Mason University(乔治梅森大学);GE Healthcare(GE医疗);University of Pittsburgh(匹兹堡大学);William and Mary(威廉玛丽学院);Southern University of Science and Technology(南方科技大学);Florida State University(佛罗里达州立大学)
SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection
img/wse-cube-no-shadow.png Web \& Software Engineering (WSE) Research Group, Leipzig University of Applied Sciences (HTWK Leipzig), Leipzig, Germany CBZ München GmbH, Heilbronn, Germany
Mitigating Shortcuts in Language Models with Soft Label Encoding
Stevens Institute of Technology(史蒂文斯理工学院);Shanghai Jiao Tong University(上海交通大学);New Jersey Institute of Technology(新泽西理工学院);University of Georgia(佐治亚大学)