Modern language models (LM) are trained on large scrapes of the Web, containing millions of personal information (PI) instances, many of which LMs memorize, increasing privacy risks. In this work, we develop the regexes and rules (R&R) detector suite to detect email addresses, phone numbers, and IP addresses, which outperforms the best regex-based PI detectors. On a manually curated set of 483 instances of PI, we measure memorization: finding that 13.6% are parroted verbatim by the Pythia-6.9b model, i.e., when the model is prompted with the tokens that precede the PI in the original document, greedy decoding generates the entire PI span exactly. We expand this analysis to study models of varying sizes (160M-6.9B) and pretraining time steps (70k-143k iterations) in the Pythia model suite and find that both model size and amount of pretraining are positively correlated with memorization. Even the smallest model, Pythia-160m, parrots 2.7% of the instances exactly. Consequently, we strongly recommend that pretraining datasets be aggressively filtered and anonymized to minimize PI parroting.
Less Is Personal: Learning Minimal Sufficient User Profiles for Personalized Language Models
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
PEToolLLM: Towards Personalized Tool Learning in Large Language Models
The Hong Kong Polytechnic University(香港理工大学);National University of Singapore(新加坡国立大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
LLMvsSmall Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model
Beijing Institute of Technology(北京理工大学);Beijing University of Posts and Telecommunications(北京邮电大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
A Natural Language Query Interface for Searching Personal Information on Smartwatches
Dartmouth College(达特茅斯学院);Liverpool John Moore University(利物浦约翰摩尔斯大学);Vienna University of Technology(维也纳工业大学);University of Vienna(维也纳大学);University of California, Riverside(加州大学河滨分校)
ReLay: Personalized LLM-Generated Plain-Language Summaries for Better Understanding, but at What Cost?
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Virginia(弗吉尼亚大学);University of Colorado Anschutz(科罗拉多大学安舒茨分校);Johns Hopkins University(约翰霍普金斯大学);University of Chicago Pritzker School of Medicine(芝加哥大学普ritzker法学院);National Library of Medicine, National Institutes of Health(国家医学图书馆,国家卫生研究院);University of Washington(华盛顿大学)