arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2206.06952cs.CLcs.AIcs.LG
收藏

FETILDA: 面向长金融文本文档的微调嵌入有效框架

FETILDA: An Effective Framework For Fin-tuned Embeddings For Long Financial Text Documents

Rensselaer Polytechnic Institute(伦斯勒理工学院);School of Science(理学院);Lally School of Management(拉里管理学院)

Bolun "Namir" Xia, Vipula D. Rawte, Mohammed J. Zaki, Aparna Gupta

AI总结:

针对金融长文本表征难题,提出FETILDA框架,通过分块处理、预训练LM聚合与自注意力提取特征,在两个金融披露数据集上优于基线,提升了文本回归预测效果。

摘要:

非结构化数据(尤其是文本)在各领域持续快速增长。在金融领域,积累了大量非结构化金融数据,例如企业定期向美国证券交易委员会(SEC)等监管机构提交的文本披露文件。这些文件通常篇幅很长,往往包含关于公司业绩的宝贵软信息。因此,从这些长文本中学习预测模型,尤其是用于预测数值型关键绩效指标(KPI),具有重要研究价值。尽管预训练语言模型(LM)在海量文本语料上的学习已取得巨大进展,但它们在长文档的有效表征方面仍存在不足。我们的工作填补了这一关键需求,即如何开发更好的模型,从长文本中提取有用信息,并学习能够利用金融与风险软信息的有效特征,以用于文本回归(预测)任务。本文提出并实现了一个深度学习框架:该框架将长文档分割为多个文本块,利用预训练LM处理这些文本块并聚合成向量表征,随后通过自注意力机制提取有价值的文档级特征。我们在美国银行的10-K公开披露报告集以及另一份美国企业提交的报告数据集上评估了该模型。总体而言,我们的框架性能优于性能强劲的文本建模基线方法,也优于仅使用数值数据的基线回归模型。我们的工作为如何利用预训练的领域特定、支持长输入的微调LM表征长文档提供了更深入的见解,能够提升文本数据的表征质量,进而助力改进预测分析。

英文摘要:

Unstructured data, especially text, continues to grow rapidly in various domains. In particular, in the financial sphere, there is a wealth of accumulated unstructured financial data, such as the textual disclosure documents that companies submit on a regular basis to regulatory agencies, such as the Securities and Exchange Commission (SEC). These documents are typically very long and tend to contain valuable soft information about a company's performance. It is therefore of great interest to learn predictive models from these long textual documents, especially for forecasting numerical key performance indicators (KPIs). Whereas there has been a great progress in pre-trained language models (LMs) that learn from tremendously large corpora of textual data, they still struggle in terms of effective representations for long documents. Our work fills this critical need, namely how to develop better models to extract useful information from long textual documents and learn effective features that can leverage the soft financial and risk information for text regression (prediction) tasks. In this paper, we propose and implement a deep learning framework that splits long documents into chunks and utilizes pre-trained LMs to process and aggregate the chunks into vector representations, followed by self-attention to extract valuable document-level features. We evaluate our model on a collection of 10-K public disclosure reports from US banks, and another dataset of reports submitted by US companies. Overall, our framework outperforms strong baseline methods for textual modeling as well as a baseline regression model using only numerical data. Our work provides better insights into how utilizing pre-trained domain-specific and fine-tuned long-input LMs in representing long documents can improve the quality of representation of textual data, and therefore, help in improving predictive analyses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑