Retrieval-augmented generation (RAG) is becoming an increasingly popular technique for integrating internal knowledge bases with large language models. In a typical RAG pipeline, three models are used, responsible for the retrieval, reranking, and generation stages. In this article, we focus on the reranking problem for the Polish language, examining the performance of rerankers and comparing their results with available retrieval models. We conduct a comprehensive evaluation of existing models and those trained by us, utilizing a benchmark of 41 diverse information retrieval tasks for the Polish language. The results of our experiments show that most models struggle with out-of-domain generalization. However, a combination of effective optimization method and a large training dataset allows for building rerankers that are both compact in size and capable of generalization. The best of our models establishes a new state-of-the-art for reranking in the Polish language, outperforming existing models with up to 30 times more parameters.
Lost in Historical Time? A Polish History Matura Benchmark for Large Language Models
Adam Mickiewicz University(亚当·密茨凯维奇大学);IDEAS Research Institute(IDEAS研究院);AMU Center for Artificial Intelligence(亚当·密茨凯维奇大学人工智能中心);Poznań University of Medical Sciences(波兹南医科大学);Maria Curie-Skłodowska University(玛丽·居里-斯克洛多夫斯卡大学);WSB Merito University(WSB梅里托大学)
Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of Amsterdam(阿姆斯特丹大学)
MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
Yonsei University(延世大学);KAIST(韩国科学技术院);OneLineAI;Barcelona Supercomputing Center(巴塞罗那超级计算中心);Artful Media;Bangladesh University of Engineering and Technology(孟加拉国工程技术大学);Carnegie Mellon University(卡内基梅隆大学)