发表机构
LG AI Research（LG AI研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对金融语言模型的前视偏差，提出PALM方法，用低秩适配器替代年度预训练，在多个PIT模型上验证其有效性，优于继续预训练。
AI 中文摘要
金融回测中使用的语言模型存在前视偏差，因为模型在研究报告期之后发布的文本上进行训练，已经观察到了其需要预测的结果。为解决此问题，时点（PIT）语言模型在按时间顺序过滤的语料库上进行预训练，并以每个日历年一个检查点的形式发布，每个检查点都有明确的截止日期。然而，每增加一年就需要一次完整的预训练运行，而这种运行是否必要从未被测试过。在本文中，我们证明年度预训练运行并非必要。我们反而将每个检查点与取代它的较新检查点进行比较，发现在相同的评估窗口上，较新的检查点得分并不更高。受此观察启发，我们提出PALM（金融语言模型的时点自适应），这是一种简单而有效的年度预训练替代方案，它在决策日期之前发布的文本上拟合低秩适配器，而不修改任何预训练权重。我们进一步发现，一个小的适配器就足以将新时间段的知识添加到旧检查点已编码的知识中，并且这优于继续预训练。我们在十年的金融新闻和多种PIT模型家族上验证了PALM，这些模型的截止日期跨越二十年，规模从1.3B到4.2B不等。代码可在以下网址获取：此HTTPS URL。
英文摘要
Language models used in financial backtests suffer from look-ahead bias, as a model trained on text published after the study period has already observed the outcomes it is asked to predict. To handle this issue, point-in-time (PIT) language models are pretrained on chronologically filtered corpora and released as one checkpoint per calendar year, each with a documented cutoff. However, each additional year costs a full pretraining run, and whether that run is necessary has never been tested. In this paper, we show that the annual pretraining run is not necessary. We instead compare each checkpoint against the newer one that replaced it, and find that the newer checkpoint scores no better on the same evaluation window. Motivated by this observation, we propose PALM (Point-in-time Adaptation for financial Language Models), a simple yet effective alternative to annual pretraining that fits a low-rank adapter on text published before the decision date without modifying any pretrained weight. We further find that a small adapter is enough to add a new period to the knowledge an old checkpoint already encodes, and that this outperforms continued pretraining. We validate PALM on a decade of financial news and on various families of PIT models, whose cutoffs span two decades and whose sizes range from 1.3 to 4.2B. Code is available at: https://github.com/seunghan96/palm.
