语言大模型 / LLM - arXivDaily 专题

2303.12513 2023-11-07 cs.CV cs.CL cs.LG 84%

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

Morris Alper, Michael Fiman, Hadar Averbuch-Elor

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted to CVPR 2023. Project webpage: https://isbertblind.github.io/

2310.19961 2023-11-01 cs.LG cs.AI 84%

ExPT: Synthetic Pretraining for Few-Shot Experimental Design

Tung Nguyen, Sudhanshu Agrawal, Aditya Grover

专题命中预训练与数据：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Comments 2023 Conference on Neural Information Processing Systems (NeurIPS)

2310.17530 2023-10-27 cs.CV cs.CL cs.LG 84%

Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models

Laura Cabello, Emanuele Bugliarello, Stephanie Brandl, Desmond Elliott

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments To appear in EMNLP 2024

2310.14369 2023-10-24 cs.LG cs.AI 84%

MoPe: Model Perturbation-based Privacy Attacks on Language Models

Marvin Li, Jason Wang, Jeffrey Wang, Seth Neel

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.AI、cs.LG

2310.03477 2023-10-06 cs.CL cs.AI 84%

Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

François Remy, Pieter Delobelle, Bettina Berendt, Kris Demuynck, Thomas Demeester

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments As first reviewed at TACL

2309.09380 2023-09-19 cs.CL cs.LG 84%

Mitigating Shortcuts in Language Models with Soft Label Encoding

Zirui He, Huiqi Deng, Haiyan Zhao, Ninghao Liu, Mengnan Du

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2308.12270 2023-08-24 cs.LG cs.AI 84%

Language Reward Modulation for Pretraining Reinforcement Learning

Ademi Adeniji, Amber Xie, Carmelo Sferrazza, Younggyo Seo, Stephen James, Pieter Abbeel

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Code available at https://github.com/ademiadeniji/lamp

2308.00624 2023-08-02 cs.CL cs.AI 84%

JIANG: Chinese Open Foundation Language Model

Qinhua Duan, Wenchao Gu, Yujia Chen, Wenxin Mao, Zewen Tian, Hui Cao

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2307.14430 2023-08-01 cs.CL cs.LG 84%

Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, Christopher Ré

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2307.04401 2023-07-11 cs.CL cs.AI 84%

Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation

Zhexin Zhang, Jiaxin Wen, Minlie Huang

专题命中预训练与数据：prompting(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments ACL 2023 Long Paper (Main Conference)

2212.10785 2023-05-30 cs.CL cs.AI 84%

SERENGETI: Massively Multilingual Language Models for Africa

Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Alcides Alcoba Inciarte

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments To appear in Findings of ACL 2023

2305.15507 2023-05-26 cs.CL cs.AI 84%

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Antonio Valerio Miceli-Barone, Fazl Barez, Ioannis Konstas, Shay B. Cohen

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments 17 pages, 5 figure, ACL 2023

2305.12281 2023-05-23 cs.CL cs.LG 84%

Lifelong Language Pretraining with Distribution-Specialized Experts

Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, Claire Cu

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICML 2023

2305.10992 2023-05-19 cs.CL cs.AI 84%

How does the task complexity of masked pretraining objectives affect downstream performance?

Atsuki Yamaguchi, Hiroaki Ozaki, Terufumi Morishita, Gaku Morio, Yasuhiro Sogawa

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ACL 2023 Findings

2209.10372 2023-05-17 cs.CL cs.AI 84%

WeLM: A Well-Read Pre-trained Language Model for Chinese

Hui Su, Xiao Zhou, Houjin Yu, Xiaoyu Shen, Yuwen Chen, Zilin Zhu, Yang Yu, Jie Zhou

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2211.00151 2023-05-09 cs.CL cs.LG 84%

A Close Look into the Calibration of Pre-trained Language Models

Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, Heng Ji

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ACL 2023 main conference. Code is available at: https://github.com/lifan-yuan/PLMCalibration

2305.01550 2023-05-03 cs.CL cs.CR cs.LG 84%

Mitigating Approximate Memorization in Language Models via Dissimilarity Learned Policy

Aly M. Kassem

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2304.14275 2023-04-28 cs.CL cs.LG 84%

What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files

Peter Meltzer, Joseph G. Lambourne, Daniele Grandi

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2304.10946 2023-04-24 cs.CL cs.LG q-bio.BM 84%

CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models

Tianhao Li, Sandesh Shetty, Advaith Kamath, Ajay Jaiswal, Xianqian Jiang, Ying Ding, Yejin Kim

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.LG

2304.08865 2023-04-19 cs.CL cs.LG 84%

Romanization-based Large-scale Adaptation of Multilingual Language Models

Sukannya Purkayastha, Sebastian Ruder, Jonas Pfeiffer, Iryna Gurevych, Ivan Vulić

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 9 pages, 5 figures

2202.07206 2023-03-17 cs.CL cs.LG 84%

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Yasaman Razeghi, Robert L. Logan, Matt Gardner, Sameer Singh

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2110.03501 2023-03-15 stat.ML cs.CL cs.LG 84%

Pretrained Language Models are Symbolic Mathematics Solvers too!

Kimia Noorbakhsh, Modar Sulaiman, Mahdi Sharifi, Kallol Roy, Pooyan Jamshidi

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2303.02995 2023-03-07 cs.CV cs.CL cs.LG 84%

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, Yongfeng Zhang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at ICLR 2023

2202.07646 2023-03-07 cs.LG cs.CL 84%

Quantifying Memorization Across Neural Language Models

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2206.07699 2023-02-20 cs.CV cs.CL cs.LG 84%

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, Jiawei Wang

专题命中预训练与数据：language model(title,abstract)；foundation model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2023

2210.01296 2023-02-17 cs.CL cs.AI 84%

Recitation-Augmented Language Models

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, Denny Zhou

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICLR 2023. v2 adds the Codex results

2302.07735 2023-02-16 cs.CL cs.AI cs.CR 84%

Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Ali Al-Kaswan, Maliheh Izadi, Arie van Deursen

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2212.14034 2022-12-29 cs.CL cs.LG 84%

Cramming: Training a Language Model on a Single GPU in One Day

Jonas Geiping, Tom Goldstein

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 22 pages, we provide code at https://github.com/JonasGeiping/cramming

2208.06049 2022-12-21 cs.CV cs.CL cs.LG 84%

MILAN: Masked Image Pretraining on Language Assisted Representation

Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, Sun-Yuan Kung

专题命中预训练与数据：pretraining(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments add new experiments and improved results. provide repo link

2202.06539 2022-12-21 cs.CR cs.CL cs.LG 84%

Deduplicating Training Data Mitigates Privacy Risks in Language Models

Nikhil Kandpal, Eric Wallace, Colin Raffel

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments ICML 2022 Camera Ready Version

1. 预训练与数据 12429 篇

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

ExPT: Synthetic Pretraining for Few-Shot Experimental Design

Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models

MoPe: Model Perturbation-based Privacy Attacks on Language Models

Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

Mitigating Shortcuts in Language Models with Soft Label Encoding

Language Reward Modulation for Pretraining Reinforcement Learning

JIANG: Chinese Open Foundation Language Model

Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation

SERENGETI: Massively Multilingual Language Models for Africa

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Lifelong Language Pretraining with Distribution-Specialized Experts

How does the task complexity of masked pretraining objectives affect downstream performance?

WeLM: A Well-Read Pre-trained Language Model for Chinese

A Close Look into the Calibration of Pre-trained Language Models

Mitigating Approximate Memorization in Language Models via Dissimilarity Learned Policy

What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files

CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models

Romanization-based Large-scale Adaptation of Multilingual Language Models

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Pretrained Language Models are Symbolic Mathematics Solvers too!

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Quantifying Memorization Across Neural Language Models

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Recitation-Augmented Language Models

Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Cramming: Training a Language Model on a Single GPU in One Day

MILAN: Masked Image Pretraining on Language Assisted Representation

Deduplicating Training Data Mitigates Privacy Risks in Language Models