语言大模型 / LLM

Do we really have to filter out random noise in pre-training data for language models?

Jinghan Ru, Yuxin Xie, Xianwei Zhuang, Yuguo Yin, Zhihui Guo, Zhiming Liu, Qianli Ren, Yuexian Zou

机构 * School of Electronic and Computer Engineering, Peking University（北京理工大学电子与计算机工程学院） ； University of Electronic Science and Technology of China（电子科技大学） ； Hong Kong University of Science and Technology（香港理工大学） ； Sichuan University（四川大学） 纠错

专题命中 预训练与数据 ：language model(title)；分类 cs.CL

SAR Object Detection with Self-Supervised Pretraining and Curriculum-Aware Sampling

Yasin Almalioglu, Andrzej Kucik, Geoffrey French, Dafni Antotsiou, Alexander Adam, Cedric Archambeau

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI

Comments Accepted to ICLR 2025 ML4RS https://ml-for-rs.github.io/iclr2025/

Mind the Prompt: Prompting Strategies in Audio Generations for Improving Sound Classification

Francesca Ronchini, Ho-Hsiang Wu, Wei-Cheng Lin, Fabio Antonacci

专题命中 预训练与数据 ：prompting(title)；分类 cs.AI

Comments Accepted at Generative Data Augmentation for Real-World Signal Processing Applications Workshop

Data Selection via Optimal Control for Language Models

Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, Minlie Huang

专题命中 预训练与数据 ：language model(title)；分类 cs.CL

Comments ICLR 2025 Oral

Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversations

Ishani Mondal, Jack W. Stokes, Sujay Kumar Jauhar, Longqi Yang, Mengting Wan, Xiaofeng Xu, Xia Song, Jennifer Neville

专题命中 预训练与数据 ：LLM(title)；分类 cs.CL

Comments 23 pages

Interpreting Neurons in Deep Vision Networks with Language Models

Nicholas Bai, Rahul A. Iyer, Tuomas Oikarinen, Akshay Kulkarni, Tsui-Wei Weng

专题命中 预训练与数据 ：language model(title)；分类 cs.LG

Unifying Prediction and Explanation in Time-Series Transformers via Shapley-based Pretraining

Qisen Cheng, Jinming Xing, Chang Xue, Xiaoran Yang

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

Comments 6 pages, Accepted to 21st IEEE CSPA 2025

Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop

Ekaterina Artemova, Akim Tsvigun, Dominik Schlechtweg, Natalia Fedorova, Konstantin Chernyshev, Sergei Tilga, Boris Obmoroshev

专题命中 预训练与数据 ：LLM(title)；分类 cs.CL

Comments To be presented at COLING 2025

Stable-TTS: Stable Speaker-Adaptive Text-to-Speech Synthesis via Prosody Prompting

Wooseok Han, Minki Kang, Changhun Kim, Eunho Yang

专题命中 预训练与数据 ：prompting(title)；分类 cs.AI

Comments Accepted by ICASSP 2025

Mitigating Instance-Dependent Label Noise: Integrating Self-Supervised Pretraining with Pseudo-Label Refinement

Gouranga Bala, Anuj Gupta, Subrat Kumar Behera, Amit Sethi

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

Efficient Neural Network Training via Subset Pretraining

Jan Spörer, Bernhard Bermeitinger, Tomas Hrycej, Niklas Limacher, Siegfried Handschuh

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

Comments To appear in KDIR 2024

Journal ref Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KDIR2024

DEREC-SIMPRO: unlock Language Model benefits to advance Synthesis in Data Clean Room

Tung Sum Thomas Kwok, Chi-hua Wang, Guang Cheng

专题命中 预训练与数据 ：language model(title)；分类 cs.LG

Geometric Self-Supervised Pretraining on 3D Protein Structures using Subgraphs

Michail Chatzianastasis, Yang Zhang, George Dasoulas, Michalis Vazirgiannis

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

LLM-based Code-Switched Text Generation for Grammatical Error Correction

Tom Potter, Zheng Yuan

专题命中 预训练与数据 ：LLM(title)；分类 cs.CL

PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretraining

Kecen Li, Chen Gong, Zhixiang Li, Yuzhong Zhao, Xinwen Hou, Tianhao Wang

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

Comments Accepted at USENIX Security 2024. The first two authors contributed equally. We communicated with the author of DPSDA and have added an explanation for why the FID scores in the DPSDA table are lower than those reported in the original paper

EuroLLM: Multilingual Language Models for Europe

Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, André F. T. Martins

专题命中 预训练与数据 ：language model(title)；分类 cs.CL

Hand-Object Interaction Pretraining from Videos

Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, Jitendra Malik

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI

SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data

Yichen Lu, Jiaqi Song, Xuankai Chang, Hengwei Bian, Soumi Maiti, Shinji Watanabe

专题命中 预训练与数据 ：language model(title)；分类 cs.CL

NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation

Jingyang Huo, Yikai Wang, Xuelin Qian, Yun Wang, Chong Li, Jianfeng Feng, Yanwei Fu

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

Comments Accepted by ECCV 2024

Neuro-BERT: Rethinking Masked Autoencoding for Self-supervised Neurological Pretraining

Di Wu, Siyuan Li, Jie Yang, Mohamad Sawan

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

Comments IEEE Journal of Biomedical and Health Informatics 2024 (camera ready)

Prompting the E-Brushes: Users as Authors in Generative AI

Yiyang Mei

专题命中 预训练与数据 ：prompting(title)；分类 cs.AI

Journal ref International Journal of Law, Ethics, and Technology 2024

Contextual Spelling Correction with Language Model for Low-resource Setting

Nishant Luitel, Nirajan Bekoju, Anand Kumar Sah, Subarna Shakya

专题命中 预训练与数据 ：language model(title)；分类 cs.CL

Comments 8 pages

GraphAlign: Pretraining One Graph Neural Network on Multiple Graphs via Feature Alignment

Zhenyu Hou, Haozhan Li, Yukuo Cen, Jie Tang, Yuxiao Dong

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

CARTE: Pretraining and Transfer for Tabular Learning

Myung Jun Kim, Léo Grinsztajn, Gaël Varoquaux

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

IT5: Text-to-text Pretraining for Italian Language Understanding and Generation

Gabriele Sarti, Malvina Nissim

专题命中 预训练与数据 ：pretraining(title)；分类 cs.CL

Comments LREC-COLING 2024. Code and checkpoints: https://github.com/gsarti/it5

Journal ref Proceedings of LREC-COLING (2024) 9422-9433

AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining

Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, Mark D. Plumbley

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI

Comments Accepted by IEEE/ACM Transactions on Audio, Speech and Language Processing. Project page is https://audioldm.github.io/audioldm2

Exploring Facial Expression Recognition through Semi-Supervised Pretraining and Temporal Modeling

Jun Yu, Zhihong Wei, Zhongpeng Cai, Gongpeng Zhao, Zerui Zhang, Yongqi Wang, Guochen Xie, Jichao Zhu, Wangyuan Zhu

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI

Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data

Antonis Antoniades, Yiyi Yu, Joseph Canzano, William Wang, Spencer LaVere Smith

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG

Comments 9 pages for main paper. 22 pages in total. 13 figures, 1 table

Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi

Md Nishat Raihan, Dhiman Goswami, Antara Mahmud

专题命中 预训练与数据 ：language model(title)；分类 cs.CL

On the Generalization Ability of Unsupervised Pretraining

Yuyang Deng, Junyuan Hong, Jiayu Zhou, Mehrdad Mahdavi

专题命中 预训练与数据 ：pretraining(title)；分类 cs.LG