Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context. At the same time, there is a lack of clear understanding about both the presence and further characteristics of this capability in deep learning models. In this paper, we introduce a systematic probing framework to explore the abstraction capability of deep learning models from a transferability perspective. A set of controlled experiments are conducted based on this framework, providing strong evidence that two probed pre-trained language models (PLMs), T5 and GPT2, have the abstraction capability. We also conduct in-depth analysis, thus shedding further light: (1) the whole training phase exhibits a "memorize-then-abstract" two-stage process; (2) the learned abstract concepts are gathered in a few middle-layer attention heads, rather than being evenly distributed throughout the model; (3) the probed abstraction capabilities exhibit robustness against concept mutations, and are more robust to low-level/source-side mutations than high-level/target-side ones; (4) generic pre-training is critical to the emergence of abstraction capability, and PLMs exhibit better abstraction with larger model sizes and data scales.
Leveraging LLMs for Title and Abstract Screening for Systematic Review: A Cost-Effective Dynamic Few-Shot Learning Approach
Duke University(杜克大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);Duke University School of Medicine(杜克大学医学院);Duke Clinical Research Institute(杜克临床研究所)
A Systematic Study of Small Language Models on Abstract Reasoning Tasks
Leibniz University of Hanover(莱布尼茨汉诺威大学);TIB – Leibniz Information Centre for Science and Technology and University Library(TIB – 莱布尼茨科学技术信息中心与大学图书馆);Amazon(亚马逊)
Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
University of California, Los Angeles(加州大学洛杉矶分校);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Tsinghua University(清华大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)
A large scale study of SVM based methods for abstract screening in systematic reviews
Indiana University Purdue University Indianapolis(印第安纳大学与普渡大学印第安纳波利斯联合分校);Qatar Computing Research Institute, Hamad-Bin Khalifa University(哈马德·本·哈利法大学卡塔尔计算研究所);University of Lille(里尔大学)
系统评价的一项主要任务是摘要筛选,即,基于标题和摘要排除从数据库检索返回的通常成百上千的不相关引文。因此,一个可以自动化摘要筛选过程的系统性综述平台非常重要。已经为这项任务提出了几种方法。然而,由于以下挑战,很难清楚地了解这些方法在系统评价平台中的适用性:(1)使用非重叠度量来评估所提出的方法,(2)使用非常难以收集的特征,(3)使用一小组评论来评估,以及(4)没有可靠的统计检验或方法的等效分组。在本文中,我们使用的特征表示,可以提取每个引用。我们评估基于SVM的方法(常用)的一个大的审查($61$)和指标($11$),以提供基于坚实的统计检验的方法的等效分组。我们的分析还包括使用$500$ x $2$交叉验证的度量的强可变性。虽然有些方法适用于不同的指标和不同的数据集,但没有一种方法可以在其中占据主导地位。此外,我们观察到,在某些情况下,通过基于确定性的抽样,仅筛选其中的15-20%后,可以找到相关(包括)引文。少数纳入的引文存在异常特征,只能在大量筛选步骤后才能找到。最后,我们提出了一个集成算法产生一个$5$星级的引文的相关性的基础上。这种算法结合了我们评估的最佳方法,并通过其$5$星级评级输出更易于消费的预测。
Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis
The Alan Turing Institute(艾伦·图灵研究所);University of Bristol(布里斯托大学);University of Manchester(曼彻斯特大学);The Institute of Cancer Research(癌症研究所);Genentech(基因泰克)