在更大规模的语言模型中通过人类上下文解决生态谬误
Addressing the Ecological Fallacy in Larger LMs with Human Context
AI总结:
通过HuLM任务建模作者语言上下文,提升大模型在多个下游任务上的性能。
摘要:
语言模型的训练和推理忽略了语言学中的一个基本事实--同一人写的多个文本序列之间存在依赖关系。先前的工作表明,通过解决这种生态谬误形式可以显著提高多个较小(~124M)基于GPT的模型的性能。在本工作中,我们问通过用特定的语言模型任务(称为HuLM)建模作者的语言上下文,是否能为更大规模的模型(一个8B的Llama模型)提供类似的收益。为此,我们探索了处理作者语言的变体,在其其他时间有序文本的背景下。我们研究了在预训练中使用此作者上下文使用HuLM目标的效果,以及在微调时使用作者上下文(称为HuFT:人类感知微调)。实证比较表明,仅在微调期间使用QLoRA解决生态谬误可以提高8B模型的性能,优于标准微调。此外,基于QLoRA的持续HuLM预训练结果产生了一个具有人类感知的模型,能够在仅使用线性任务分类器训练的情况下,在八个下游任务上实现改进的性能。这些结果表明,建模语言在原始生成者(作者)的上下文中的重要性和实用性。
英文摘要:
Language model training and inference ignore a fundamental linguistic fact -- there is a dependence between multiple sequences of text written by the same person. Prior work has shown that addressing this form of \textit{ecological fallacy} can greatly improve the performance of multiple smaller (~124M) GPT-based models. In this work, we ask if addressing the ecological fallacy by modeling the author's language context with a specific LM task (called HuLM) can provide similar benefits for a larger-scale model, an 8B Llama model. To this end, we explore variants that process an author's language in the context of their other temporally ordered texts. We study the effect of pre-training with this author context using the HuLM objective, as well as using it during fine-tuning with author context (\textit{HuFT:Human-aware Fine-Tuning}). Empirical comparisons show that addressing the ecological fallacy during fine-tuning alone using QLoRA improves the performance of the larger 8B model over standard fine-tuning. Additionally, QLoRA-based continued HuLM pre-training results in a human-aware model generalizable for improved performance over eight downstream tasks with linear task classifier training alone. These results indicate the utility and importance of modeling language in the context of its original generators, the authors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。