arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.01015cs.CVcs.AIcs.CL
收藏

半结构化对象序列编码器

Semi-Structured Object Sequence Encoders

IBM Research AI(IBM研究院人工智能部门);Microsoft India Development Center(微软印度开发中心)

Rudra Murthy, Riyaz Bhat, Chulaka Gunasekara, Siva Sankalp Patel, Hui Wan, Tejas Indulal Dhamecha, Danish Contractor, Marina Danilevsky

AI总结:

本文研究半结构化对象序列的建模问题,提出一种两阶段方法:先独立编码各键的值表示,再通过共享注意力头架构进行自注意力操作,以处理更长的序列并在多个真实预测任务中优于分层编码等基线方法。

摘要:

在本文中,我们探索了半结构化对象序列的建模任务;特别地,我们将注意力集中在为这类序列开发结构感知的输入表示问题上。此类数据的例子包括网站上的用户活动、机器日志等。这类数据通常表示为随时间变化的键值对集合序列,由于序列长度不断增加,可能会带来建模挑战。我们提出了一种分为两部分的方法,首先独立考虑每个键,并随时间编码其值的表示;然后对这些值感知的键表示进行自注意力操作以完成下游任务。这使我们能够处理比现有方法更长的对象序列。我们在两个模块之间引入了一种新颖的共享注意力头架构,并提出了一种创新的训练计划,该计划将两个模块的训练与部分注意力头的共享权重交替进行。我们在使用真实世界数据的多个预测任务上的实验表明,我们的方法优于具有分层编码的统一网络,以及其他方法,包括以记录为中心的表示和序列的扁平化表示。

英文摘要:

In this paper we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input representation for such sequences. Examples of such data include user activity on websites, machine logs, and many others. This type of data is often represented as a sequence of sets of key-value pairs over time and can present modeling challenges due to an ever-increasing sequence length. We propose a two-part approach, which first considers each key independently and encodes a representation of its values over time; we then self-attend over these value-aware key representations to accomplish a downstream task. This allows us to operate on longer object sequences than existing methods. We introduce a novel shared-attention-head architecture between the two modules and present an innovative training schedule that interleaves the training of both modules with shared weights for some attention heads. Our experiments on multiple prediction tasks using real-world data demonstrate that our approach outperforms a unified network with hierarchical encoding, as well as other methods including a record-centric representation and a flattened representation of the sequence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑