The popularization of social media increases user engagements and generates a large amount of user-oriented data. Among them, text data (e.g., tweets, blogs) significantly attracts researchers and speculators to infer user attributes (e.g., age, gender, location) for fulfilling their intents. Generally, this line of work casts attribute inference as a text classification problem, and starts to leverage graph neural networks (GNNs) to utilize higher-level representations of source texts. However, these text graphs are constructed over words, suffering from high memory consumption and ineffectiveness on few labeled texts. To address this challenge, we design a text-graph-based few-shot learning model for attribute inferences on social media text data. Our model first constructs and refines a text graph using manifold learning and message passing, which offers a better trade-off between expressiveness and complexity. Afterwards, to further use cross-domain texts and unlabeled texts to improve few-shot performance, a hierarchical knowledge distillation is devised over text graph to optimize the problem, which derives better text representations, and advances model generalization ability. Experiments on social media datasets demonstrate the state-of-the-art performance of our model on attribute inferences with considerably fewer labeled texts.
Hierarchical Multi-to-Single-Modal Knowledge Distillation for Disruption Prediction in EAST
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Institute of Plasma Physics, Chinese Academy of Sciences（中国科学院等离子体物理研究所）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University（安徽大学光电子信息获取与控制技术国家重点实验室）
Debiased Distillation by Transplanting the Last Layer
Department of Electrical Engineering, POSTECH（POSTECH电气工程系）；Graduate School of Artificial Intelligence, POSTECH（POSTECH人工智能研究生院）；POSTECH（POSTECH（浦项科技大学））
深度模型很容易学习虚假的相关性，即使在后处理过程中也是如此。我们仔细研究了知识蒸馏--一种流行的模型压缩后处理技术--并发现，使用有偏见的训练数据进行蒸馏会产生有偏见的学生，即使教师是无偏见的。为了解决这个问题，我们提出了一个简单的知识蒸馏算法，创造DeTT（Debiasing by Teacher Transplanting）。最近的一项观察发现，最后一层神经网络在去偏置中起着极其重要的作用，受此启发，DeTT将教师的最后一层直接移植到学生身上。通过匹配学生和教师的特征图输出来提取剩余的层，其中样本被重新加权以减轻数据集偏差。重要的是，DeTT不依赖于偏差相关属性上的广泛注释的可用性，这通常在后处理阶段不可用。在我们的整个实验中，DeTT成功地消除了学生模型的偏差，在最差组准确性方面始终优于基线。
Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation
Chongqing University of Technology（重庆理工大学）；Fuzhou University（福州大学）；China Automotive Engineering Research Institute Co., Ltd.（中国汽车工程研究院股份有限公司）；Chongqing Changan Automobile Co., Ltd.（重庆长安汽车股份有限公司）
Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation
Ant Group（蚂蚁集团）；Alibaba International Digital Commerce Group（阿里巴巴国际数字商业集团）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；University of Electronic Science and Technology of China（电子科技大学）
