arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.11045cs.CVcs.AI
收藏

通过合成数据集设计研究神经结构

Investigating Neural Architectures by Synthetic Dataset Design

Centre Borelli(博雷利中心);ENS Paris-Saclay(巴黎萨克雷高等师范学院)

Adrien Courtois, Jean-Michel Morel, Pablo Arias

摘要:

近年来,出现了许多新的神经网络结构(架构和层)。为了解决一个给定的任务,网络需要在其结构中反映出一定的能力。所需的能力取决于每项任务。到目前为止,还没有系统的研究所提出的神经结构的实际能力。每个结构能够和不能实现什么的问题只能部分地通过其在共同基准上的表现来回答。事实上,自然数据包含复杂的未知统计线索。因此,不可能知道给定的神经结构在这些数据中利用了什么线索。在这项工作中,我们勾勒出一种方法来衡量每个结构对网络的能力的影响,通过设计特设的合成数据集。每个数据集都是为评估给定的能力而定制的,并被简化为最简单的形式:每个输入都包含解决任务所需的信息量。我们通过构建三个数据集来评估以下三个网络属性中的每一个来说明我们的方法:a)将本地线索与远距离推断联系起来的能力,b)平移协方差和c)将具有相同特征的像素分组并在其中共享信息的能力。使用第一个简化的深度估计数据集,我们查明了严重的非本地赤字的U-网。然后,我们评估如何通过将其结构嵌入非局部层来解决这一限制,这允许计算具有远程依赖关系的复杂特征。使用第二个数据集,我们比较了不同的位置编码方法,并使用结果来进一步改进深度估计任务上的U-Net。第三个介绍的数据集用于证明需要自我注意的机制,解决更现实的深度估计任务。

英文摘要:

Recent years have seen the emergence of many new neural network structures (architectures and layers). To solve a given task, a network requires a certain set of abilities reflected in its structure. The required abilities depend on each task. There is so far no systematic study of the real capacities of the proposed neural structures. The question of what each structure can and cannot achieve is only partially answered by its performance on common benchmarks. Indeed, natural data contain complex unknown statistical cues. It is therefore impossible to know what cues a given neural structure is taking advantage of in such data. In this work, we sketch a methodology to measure the effect of each structure on a network's ability, by designing ad hoc synthetic datasets. Each dataset is tailored to assess a given ability and is reduced to its simplest form: each input contains exactly the amount of information needed to solve the task. We illustrate our methodology by building three datasets to evaluate each of the three following network properties: a) the ability to link local cues to distant inferences, b) the translation covariance and c) the ability to group pixels with the same characteristics and share information among them. Using a first simplified depth estimation dataset, we pinpoint a serious nonlocal deficit of the U-Net. We then evaluate how to resolve this limitation by embedding its structure with nonlocal layers, which allow computing complex features with long-range dependencies. Using a second dataset, we compare different positional encoding methods and use the results to further improve the U-Net on the depth estimation task. The third introduced dataset serves to demonstrate the need for self-attention-like mechanisms for resolving more realistic depth estimation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑