APAR：通过算术感知预训练与自适应正则化微调对表格回归中的不规则目标函数进行建模
APAR: Modeling Irregular Target Functions in Tabular Regression via Arithmetic-Aware Pre-Training and Adaptive-Regularized Fine-Tuning
AI总结：
本文提出APAR框架，通过算术感知预训练和自适应正则化微调，有效拟合表格回归中的不规则目标函数，并在10个数据集上显著优于现有方法，RMSE提升9.43%至20.37%。
摘要：
表格数据在常见的机器学习应用中至关重要，涵盖金融、基因组学和医疗保健等领域。本文聚焦于表格回归任务，在该领域中，由于表格数据固有的不规则目标函数带来的挑战，导致特征微小变化即可引起标签敏感变动，深度学习方法并不始终优于机器学习模型。为解决这些问题，我们提出了一种新颖的算术感知预训练与自适应正则化微调框架（APAR），该框架使模型能够拟合表格数据中的不规则目标函数，同时减少过拟合的负面影响。在预训练阶段，APAR引入了一个算术感知的预文本目标，从连续标签的角度捕捉复杂的样本间关系。在微调阶段，提出了一种基于一致性的自适应正则化技术，以自主学习合适的数据增强。在10个数据集上的大量实验表明，APAR在RMSE指标上优于现有的基于GBDT、监督神经网络以及预训练-微调神经网络的方法（提升幅度为+9.43%至20.37%），并实证验证了预训练任务的效果，包括对算术运算的研究。我们的代码和数据已在https://github.com/johnnyhwu/APAR公开提供。
英文摘要：
Tabular data are fundamental in common machine learning applications, ranging from finance to genomics and healthcare. This paper focuses on tabular regression tasks, a field where deep learning (DL) methods are not consistently superior to machine learning (ML) models due to the challenges posed by irregular target functions inherent in tabular data, causing sensitive label changes with minor variations from features. To address these issues, we propose a novel Arithmetic-Aware Pre-training and Adaptive-Regularized Fine-tuning framework (APAR), which enables the model to fit irregular target function in tabular data while reducing the negative impact of overfitting. In the pre-training phase, APAR introduces an arithmetic-aware pretext objective to capture intricate sample-wise relationships from the perspective of continuous labels. In the fine-tuning phase, a consistency-based adaptive regularization technique is proposed to self-learn appropriate data augmentation. Extensive experiments across 10 datasets demonstrated that APAR outperforms existing GBDT-, supervised NN-, and pretrain-finetune NN-based methods in RMSE (+9.43% $\sim$ 20.37%), and empirically validated the effects of pre-training tasks, including the study of arithmetic operations. Our code and data are publicly available at https://github.com/johnnyhwu/APAR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
