Downstream-Pretext Domain Knowledge Traceback for Active Learning
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Science and Technology of China（中国科学技术大学）；University of Rochester（罗切斯特大学）
Active learning (AL) is designed to construct a high-quality labeled dataset by iteratively selecting the most informative samples. Such sampling heavily relies on data representation, while recently pre-training is popular for robust feature learning. However, as pre-training utilizes low-level pretext tasks that lack annotation, directly using pre-trained representation in AL is inadequate for determining the sampling score. To address this problem, we propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance for selecting diverse and instructive samples near the decision boundary. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. The diversity indicator constructs two feature spaces based on the pre-training pretext model and the downstream knowledge from annotation, by which it locates the neighbors of unlabeled data from the downstream space in the pretext space to explore the interaction of samples. With this mechanism, DOKT unifies the data relations of low-level and high-level representations to estimate traceback diversity. Next, in the uncertainty estimator, domain mixing is designed to enforce perceptual perturbing to unlabeled samples with similar visual patches in the pretext space. Then the divergence of perturbed samples is measured to estimate the domain uncertainty. As a result, DOKT selects the most diverse and important samples based on these two modules. The experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods and generalizes well to various application scenarios such as semantic segmentation and image captioning.
Self-Supervised Pretext Tasks for Alzheimer's Disease Classification using 3D Convolutional Neural Networks on Large-Scale Synthetic Neuroimaging Dataset
学习有意义的蛋白质表示对于各种生物下游任务（如基于结构的药物设计）非常重要。在蛋白质序列预训练取得成功之后，对信息量更大的结构数据进行预训练成为一个很有前途的研究课题。然而，蛋白质结构预训练面临三大挑战：样本多样性不足，物理上不切实际的建模，以及缺乏蛋白质特定的借口任务。为了解决这些挑战，我们提出了3D几何预训练。在本文中，我们提出了一个统一的蛋白质预训练框架和一个基于3D几何的，数据高效的，蛋白质特定的借口任务：RefineDiff（Refine the Diffused Protein Structure Decoy）。在有限的数据（不到SOTA模型的1%）上对我们的几何感知模型进行预训练后，我们获得了信息丰富的蛋白质表示，可以在各种下游任务中实现相当的性能。
Self-Pro: A Self-Prompt and Tuning Framework for Graph Neural Networks
School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；School of Computer and Information Engineering, Shanghai Polytechnic University（上海第二工业大学计算机与信息工程学院）
