Multi-task learning (mtl) provides state-of-the-art results in many applications of computer vision and natural language processing. In contrast to single-task learning (stl), mtl allows for leveraging knowledge between related tasks improving prediction results on the main task (in contrast to an auxiliary task) or all tasks. However, there is a limited number of comparative studies on applying mtl architectures for regression and time series problems taking recent advances of mtl into account. An interesting, non-linear problem is the forecast of the expected power generation for renewable power plants. Therefore, this article provides a comparative study of the following recent and important mtl architectures: Hard parameter sharing, cross-stitch network, sluice network (sn). They are compared to a multi-layer perceptron model of similar size in an stl setting. Additionally, we provide a simple, yet effective approach to model task specific information through an embedding layer in an multi-layer perceptron, referred to as task embedding. Further, we introduce a new mtl architecture named emerging relation network (ern), which can be considered as an extension of the sluice network. For a solar power dataset, the task embedding achieves the best mean improvement with 14.9%. The mean improvement of the ern and the sn on the solar dataset is of similar magnitude with 14.7% and 14.8%. On a wind power dataset, only the ern achieves a significant improvement of up to 7.7%. Results suggest that the ern is beneficial when tasks are only loosely related and the prediction problem is more non-linear. Contrary, the proposed task embedding is advantageous when tasks are strongly correlated. Further, the task embedding provides an effective approach with reduced computational effort compared to other mtl architectures.
CORE: A Knowledge Graph Entity Type Prediction Method via Complex Space Regression and Embedding
University of Southern California(南加州大学);National University of Singapore(新加坡国立大学)
实体类型预测是知识图谱研究中的一个重要问题。提出了一种新的KG实体类型预测方法CORE(Complex space Regression and Embedding)。所提出的CORE方法利用了两个复杂空间嵌入模型的表达能力,即RotatE和ComplEx模型。它使用RotatE或ComplEx在两个不同的复杂空间中嵌入实体和类型。然后,我们推导出一个复回归模型来连接这两个空间。最后,介绍了一种机制,以优化嵌入和回归参数联合。实验表明,CORE优于基准测试方法在代表KG实体类型推断数据集。分析了各种实体类型预测方法的优缺点。
VN Network: Embedding Newly Emerging Entities with Virtual Neighbors
University of Chinese Academy of Sciences(中国科学院大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Shandong University(山东大学);Tencent AI Lab(腾讯AI实验室)
Data-Efficient Symbolic Regression via Foundation Model Distillation
Institute for Clarity in Documentation(文档清晰研究所);Inria Paris-Rocquencourt(巴黎-罗克琴特研究所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室);Arizona State University(亚利桑那州立大学);Clemson University(克莱姆森大学)
Nonlinear functional regression by functional deep neural network with kernel embedding
Department of Statistics and Actuarial Science, University of Hong Kong(香港大学统计与精算科学系);Department of Mathematics, Hong Kong Baptist University(香港 Baptist 大学数学系);School of Mathematics and Statistics, University of Sydney(悉尼大学数学与统计学学院);Department of Electrical Engineering, ESAT-STADIUS, KU Leuven(KU Leuven 电子工程系)
Emerging-properties Mapping Using Spatial Embedding Statistics: EMUSES
Groupe d'Imagerie Neurofonctionnelle (GIN), Institut des Maladies Neurodegeneratives- UMR 5293, CNRS(法国国家科学研究中心神经退行性疾病研究所神经功能成像组);Brain Connectivity and Behaviour Laboratory, Sorbonne Universities(索邦大学脑连接与行为实验室);Stanford Medical School(斯坦福医学院);University of Padova(帕多瓦大学);Padova Neuroscience Center (PNC), University of Padova(帕多瓦大学帕多瓦神经科学中心);Venetian Institute of Molecular Medicine, VIMM(威尼托分子医学研究所)
Embedding interpretable $\ell_1$-regression into neural networks for uncovering temporal structure in cell imaging
Institute of Medical Biometry and Statistics (IMBI), Faculty of Medicine and Medical Center, University of Freiburg, Germany(医学生物度量与统计研究所(IMBI)、医学学院与医疗中心、弗赖堡大学);Institute of Physiology I, Faculty of Medicine, University of Freiburg, Germany(医学生理学研究所I、医学学院、弗赖堡大学);Department of Computer Science, University of Freiburg, Germany(计算机科学系、弗赖堡大学);Freiburg Center for Data Analysis, Modeling and AI (FDMAI), University of Freiburg, Germany(弗赖堡数据分析、建模与人工智能中心(FDMAI)、弗赖堡大学);Centre for Integrative Biological Signalling Studies (CIBSS), University of Freiburg, Germany(整合生物信号研究中心(CIBSS)、弗赖堡大学)
Predicted Embedding Power Regression for Large-Scale Out-of-Distribution Detection
Rochester Institute of Technology(罗切斯特理工学院)
分发外(OOD)输入可能会损害现实世界机器学习系统的性能和安全性。虽然存在许多用于OOD检测的方法,并且在具有较低分辨率和较少类别的小规模数据集上工作良好,但是很少有方法被开发用于大规模OOD检测。现有的大规模方法通常依赖于最大分类概率,例如最先进的分组softmax方法。在这项工作中,我们开发了一种新的方法,该方法基于在训练过程中学习到的标签分布来计算预测类标签的概率。我们的方法比当前最先进的方法性能更好,计算成本只有微不足道的增加。我们在$14$数据集上对我们的方法进行了评估,并在AUROC(84.2 vs 82.4)和AUPR(96.2 vs 93.7)方面取得了统计学显著的改善。
Verifiable Regularity Criterion for Conditional Expectation Operators and Conditional Mean Embeddings with Applications to Nonparametric Regression, Bayesian Inverse Problems, and Koopman Operators
Technische Universität Ilmenau(伊尔梅瑙工业大学);Freie Universität Berlin(柏林自由大学);Chemnitz University of Technology(开姆尼茨工业大学)
Demographic Predictability in 3D CT Foundation Embeddings
Rice University(莱斯大学);McGovern Medical School, UTHealth Houston(德克萨斯大学休斯顿健康科学中心麦戈文医学院);Johns Hopkins University School of Medicine(约翰霍普金斯大学医学院);University of Maryland School of Medicine(马里兰大学医学院)
Learning Output Embeddings in Structured Prediction
LTCI, Télécom Paris, IP Paris(LTCI,巴黎电信,巴黎理工学院);INRIA - Département d’Informatique de l’ENS, PSL(法国国家信息与自动化研究所 - 巴黎高等师范学院计算机系,巴黎文理研究大学);INRAE, MIAT, Université de Toulouse(法国国家农业食品与环境研究院,图卢兹大学);Department of Computer Science, Aalto University(阿尔托大学计算机科学系)
Theoretical and Experimental Analysis on the Generalizability of Distribution Regression Network
National University of Singapore(新加坡国立大学);Bioinformatics Institute, A*STAR(A*STAR生物信息学研究所)
人们对在分布之间进行回归产生了兴趣。与对单个实例的预测相反,这些机器学习方法对于基于人群的研究或本质上具有统计性质的问题可能是有用的。与传统的神经网络相比,最近提出的分布回归网络(DRN)在分布到分布的回归任务中表现出了优异的性能。然而,在Kou et al.(2018)和其他一些关于分布回归的工作中,缺乏对方法的理论基础和推广能力的全面比较研究。我们推导了DRN的一些数学性质,并将其与传统的神经网络进行了定性比较。我们还进行了全面的实验,研究分布回归模型的泛化能力,通过研究他们的鲁棒性有限的训练数据,数据采样噪声和任务难度。DRN始终优于传统的神经网络,需要更少的训练数据,并保持稳健的性能与噪声。此外,DRN的理论性质可以用来提供一些解释的能力,DRN实现更好的泛化性能比传统的神经网络。
Understanding graph embedding methods and their applications
McGovern Institute for Brain Research, Massachusetts Institute of Technology(麻省理工学院麦戈文脑科学研究所);Division of Applied Mathematics, Brown University(布朗大学应用数学系)
An FEA surrogate model with Boundary Oriented Graph Embedding approach
Purdue University(普渡大学);Nanjing Tech University(南京工业大学);Indiana Manufacturing Competitiveness Center (IN-MaC)(印第安纳制造竞争力中心)
在这项工作中,我们提出了一个边界定向图嵌入(BOGE)方法的图形神经网络(GNN)作为一个通用的代理模型回归物理场和解决边值问题。BOGE方法为边界元和局部相邻单元提供了快捷方式,可以将结构化网格单元嵌入到图中,并对基于三角形网格的大规模有限元分析结果进行有效的回归,这是其他基于机器学习的代理方法无法实现的。针对悬臂梁问题,BOGE方法不仅能拟合应力场分布,还能回归拓扑优化结果,显示了其实现抽象决策设计过程的潜力。采用3层DeepGCN模型的BOGE方法\textcolor{blue}{achieves the regression with MSE of 0.011706 (2.41\% MAPE) for stress field prediction and 0.002735 MSE (with 1.58\% elements having error larger than 0.01) for topological optimization.} BOGE方法的总体概念为通用且高效的基于深度学习的FEA模拟器铺平了道路,这将使行业和设计相关领域受益。