语言大模型 / LLM - arXivDaily 专题

2204.09653 2022-04-21 cs.PL cs.CL cs.SE 79%

On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages

Fuxiang Chen, Fatemeh Fard, David Lo, Timofey Bryksin

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted in ICPC 2022

2204.06130 2022-04-21 cs.CL 79%

Impossible Triangle: What's Next for Pre-trained Language Models?

Chenguang Zhu, Michael Zeng

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

2110.08547 2022-04-14 cs.CL 79%

Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation

Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.CL

Comments Accepted to ACL 2022. Code will be available at [https://github.com/ghchen18/acl22-sixtp]

2204.05356 2022-04-13 cs.CL 79%

A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis

Ehsan Hosseini-Asl, Wenhao Liu, Caiming Xiong

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted to Findings of NAACL 2022

2204.01440 2022-04-05 cs.CL cs.CY 79%

Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Study

Serra Sinem Tekiroglu, Helena Bonaldi, Margherita Fanton, Marco Guerini

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments To appear in "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL): Findings"

2109.07323 2022-03-29 cs.IR cs.LG 79%

FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining

Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng, Dongmei Zhang

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

Comments Accepted by ACL'22 main track

2203.12788 2022-03-25 cs.CL 79%

Evaluating Distributional Distortion in Neural Language Modeling

Benjamin LeBrun, Alessandro Sordoni, Timothy J. O'Donnell

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Journal ref International Conference on Learning Representations. 2022

2203.08928 2022-03-23 cs.CL 79%

C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References

Xiang Yue, Xiaoman Pan, Wenlin Yao, Dian Yu, Dong Yu, Jianshu Chen

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.CL

Comments ACL 2022 Main Conference

2203.11431 2022-03-23 cs.CL 79%

Task-guided Disentangled Tuning for Pretrained Language Models

Jiali Zeng, Yufan Jiang, Shuangzhi Wu, Yongjing Yin, Mu Li

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Findings of ACL 2022

2203.10753 2022-03-22 cs.CL 79%

Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

Yoshinari Fujinuma, Jordan Boyd-Graber, Katharina Kann

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.CL

Comments ACL 2022 camera ready

2110.06852 2022-03-22 cs.CL 79%

Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects

Go Inoue, Salam Khalifa, Nizar Habash

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted to Findings of ACL 2022

2108.13655 2022-03-21 cs.CL 79%

MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER

Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cambria, Luo Si, Chunyan Miao

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted at ACL 2022

2110.08484 2022-03-16 cs.CV cs.CL 79%

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted to ACL 2022

2110.08443 2022-03-10 cs.CL 79%

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Wenxuan Zhou, Fangyu Liu, Ivan Vulić, Nigel Collier, Muhao Chen

专题命中预训练与数据：pretraining(title)；language model(abstract)；分类 cs.CL

Comments ACL 2022

2203.03582 2022-03-08 cs.CL cs.SD eess.AS 79%

Improving CTC-based speech recognition via knowledge transferring from pre-trained language models

Keqi Deng, Songjun Cao, Yike Zhang, Long Ma, Gaofeng Cheng, Ji Xu, Pengyuan Zhang

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments ICASSP 2022

2112.01529 2022-03-04 cs.CV cs.LG 79%

BEVT: BERT Pretraining of Video Transformers

Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, Lu Yuan

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

Comments To Appear at CVPR 2022, code is available at https://github.com/xyzforever/BEVT

2202.09955 2022-02-24 cs.CL 79%

StyleBERT: Chinese pretraining by font style information

Chao Lv, Han Zhang, XinKai Du, Yunhao Zhang, Ying Huang, Wenhao Li, Jia Han, Shanshan Gu

专题命中预训练与数据：pretraining(title)；language model(abstract)；分类 cs.CL

2202.09452 2022-02-22 cs.CL 79%

From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French

Simon Gabay, Pedro Ortiz Suarez, Alexandre Bartz, Alix Chagué, Rachel Bawden, Philippe Gambette, Benoît Sagot

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 8 pages, 2 figures, 4 tables

2202.03514 2022-02-09 cs.SD cs.LG eess.AS 79%

Maximizing Audio Event Detection Model Performance on Small Datasets Through Knowledge Transfer, Data Augmentation, And Pretraining: An Ablation Study

Daniel Tompkins, Kshitiz Kumar, Jian Wu

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

2201.11990 2022-02-07 cs.CL 79%

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Shaden Smith and Mostofa Patwary contributed equally

2201.08643 2022-01-24 cs.CL 79%

Text Style Transfer for Bias Mitigation using Masked Language Modeling

Ewoenam Kwaku Tokpo, Toon Calders

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 9 pages, 3 figures, 5 tables

2201.05601 2022-01-19 cs.CL 79%

A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models

Vésteinn Snæbjarnarson, Haukur Barri Símonarson, Pétur Orri Ragnarsson, Svanhvít Lilja Ingólfsdóttir, Haukur Páll Jónsson, Vilhjálmur Þorsteinsson, Hafsteinn Einarsson

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

2112.08653 2021-12-17 cs.CL 79%

Reconsidering the Past: Optimizing Hidden States in Language Models

Davis Yoshida, Kevin Gimpel

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Findings of EMNLP version

Journal ref Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4099-4105

2112.07074 2021-12-15 cs.CV cs.LG 79%

Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text

Qing Li, Boqing Gong, Yin Cui, Dan Kondratyuk, Xianzhi Du, Ming-Hsuan Yang, Matthew Brown

专题命中预训练与数据：foundation model(title,abstract)；分类 cs.LG

Comments preliminary work

2112.06327 2021-12-14 cs.CL 79%

Improving Code-switching Language Modeling with Artificially Generated Texts using Cycle-consistent Adversarial Networks

Chia-Yu Li, Ngoc Thang Vu

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 4 pages, 1 figure, Interspeech 2020

2109.04650 2021-11-30 cs.CL 79%

What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Accepted to EMNLP2021 as a long paper. Fixed some typos

2101.12727 2021-11-30 cs.CV cs.LG 79%

Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency

Samarth Mishra, Kate Saenko, Venkatesh Saligrama

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.LG

Comments Accepted to BMVC 2021

2111.11646 2021-11-25 cs.CV cs.AI q-bio.QM 79%

CytoImageNet: A large-scale pretraining dataset for bioimage transfer learning

Stanley Bryan Z. Hua, Alex X. Lu, Alan M. Moses

专题命中预训练与数据：pretraining(title,abstract)；分类 cs.AI

Comments Accepted paper at NeurIPS 2021 Learning Meaningful Representations for Life (LMRL) Workshop

2106.10328 2021-11-24 cs.CL cs.CY 79%

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

Irene Solaiman, Christy Dennison

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments Both authors contributed equally. Accepted at NeurIPS 2021

2111.09509 2021-11-19 cs.CL 79%

How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN

R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, Asli Celikyilmaz

专题命中预训练与数据：language model(title,abstract)；分类 cs.CL

Comments 10 pages, plus 39 pages of appendices

1. 预训练与数据 12403 篇

On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages

Impossible Triangle: What's Next for Pre-trained Language Models?

Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation

A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis

Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Study

FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining

Evaluating Distributional Distortion in Neural Language Modeling

C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References

Task-guided Disentangled Tuning for Pretrained Language Models

Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects

MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Improving CTC-based speech recognition via knowledge transferring from pre-trained language models

BEVT: BERT Pretraining of Video Transformers

StyleBERT: Chinese pretraining by font style information

From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French

Maximizing Audio Event Detection Model Performance on Small Datasets Through Knowledge Transfer, Data Augmentation, And Pretraining: An Ablation Study

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Text Style Transfer for Bias Mitigation using Masked Language Modeling

A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models

Reconsidering the Past: Optimizing Hidden States in Language Models

Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text

Improving Code-switching Language Modeling with Artificially Generated Texts using Cycle-consistent Adversarial Networks

What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency

CytoImageNet: A large-scale pretraining dataset for bioimage transfer learning

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN