Pre-trained Language Models (PLMs), as parametric-based eager learners, have become the de-facto choice for current paradigms of Natural Language Processing (NLP). In contrast, k-Nearest-Neighbor (kNN) classifiers, as the lazy learning paradigm, tend to mitigate over-fitting and isolated noise. In this paper, we revisit kNN classifiers for augmenting the PLMs-based classifiers. From the methodological level, we propose to adopt kNN with textual representations of PLMs in two steps: (1) Utilize kNN as prior knowledge to calibrate the training process. (2) Linearly interpolate the probability distribution predicted by kNN with that of the PLMs' classifier. At the heart of our approach is the implementation of kNN-calibrated training, which treats predicted results as indicators for easy versus hard examples during the training process. From the perspective of the diversity of application scenarios, we conduct extensive experiments on fine-tuning, prompt-tuning paradigms and zero-shot, few-shot and fully-supervised settings, respectively, across eight diverse end-tasks. We hope our exploration will encourage the community to revisit the power of classical methods for efficient NLP. Code and datasets are available in https://github.com/zjunlp/Revisit-KNN.
Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
Sun Yat-sen University(中山大学);University of Cambridge(剑桥大学);University of Glasgow(格拉斯哥大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
Michigan State University(密歇根州立大学);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);UT Austin(德克萨斯大学奥斯汀分校);University of Minnesota Twin Cities(明尼苏达大学双城分校);IBM Research(IBM研究院);Princeton University(普林斯顿大学);DAMO Academy, Alibaba Group(阿里巴巴达摩院);MIT(麻省理工学院);Harvard University(哈佛大学)
SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Amazon(亚马逊);University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校);University of Texas at Austin(德克萨斯大学奥斯汀分校);University at Buffalo(布法罗大学);Northeastern University(东北大学)
Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases
Fudan University(复旦大学);School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Data Science, Fudan University(复旦大学数据科学学院);Fudan-Aishu Cognitive Intelligence Joint Research Center(复旦-爱数认知智能联合研究中心)
Revisiting Self-Training for Few-Shot Learning of Language Model
National University of Singapore(新加坡国立大学);Southern University of Science and Technology(南方科技大学);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));Kriston AI Lab(Kriston人工智能实验室)
From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
University of Chinese Academy of Sciences(中国科学院大学);Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua University(计算机科学与技术系,人工智能研究院,清华大学);Gaoling School of Artificial Intelligence, Renmin University of China(人工智能学院,中国人民大学);Kuaishou Technology Inc.(快手科技有限公司);Shanghai Key Laboratory of Multi. Info. Processing, East China Normal University(多信息处理重点实验室,华东师范大学)