Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate fine-tuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset. We observe a great variability in effectiveness when fine-tuning on different randomly selected subsets of training data. This suggests that it is possible to achieve effectiveness gains by actively selecting a subset of the training data that has the most positive effect on the rankers. This way, it would be possible to fine-tune effective PLM rankers at a reduced annotation budget. To investigate this, we adapt existing Active Learning (AL) strategies to the task of fine-tuning PLM rankers and investigate their effectiveness, also considering annotation and computational costs. Our extensive analysis shows that AL strategies do not significantly outperform random selection of training subsets in terms of effectiveness. We further find that gains provided by AL strategies come at the expense of more assessments (thus higher annotation costs) and AL strategies underperform random selection when comparing effectiveness given a fixed annotation cost. Our results highlight that ``optimal'' subsets of training data that provide high effectiveness at low annotation cost do exist, but current mainstream AL strategies applied to PLM rankers are not capable of identifying them.
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
Department of Data Science & AI, Monash University(数据科学与人工智能系,莫纳什大学);Faculty of Law, Monash University(法学院,莫纳什大学)
Modelling and Classifying the Components of a Literature Review
Knowledge Media Institute, The Open University(开放大学知识媒体研究所);The Open University(开放大学);Department of Business and Law, University of Milano Bicocca(米兰Bicocca大学商学院)
Language Models as Continuous Self-Evolving Data Engineers
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);China Mobile Internet Company Limited(中国移动互联网有限公司);Alibaba Group(阿里巴巴集团)
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
the University of Hong Kong(香港大学);Tsinghua University(清华大学);Georgia Institute of Technology(佐治亚理工学院);National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);Renmin University of China(中国人民大学);the Chinese University of Hong Kong(香港中文大学)
GaiaFlow: Semantic-Guided Diffusion Tuning for Carbon-Frugal Search
University of Macau(澳门大学);Renmin University of China(中国人民大学);Fudan University(复旦大学);Tsinghua University(清华大学);Zhejiang University(浙江大学);University of Liverpool(利物浦大学);Peking University(北京大学);Northeast Normal University(东北师范大学);Minzu University of China(民族大学)