Assessing the capabilities and limitations of large language models (LLMs) has garnered significant interest, yet the evaluation of multiple models in real-world scenarios remains rare. Multilingual evaluation often relies on translated benchmarks, which typically do not capture linguistic and cultural nuances present in the source language. This study provides an extensive assessment of 24 LLMs on real world data collected from Indian patients interacting with a medical chatbot in Indian English and 4 other Indic languages. We employ a uniform Retrieval Augmented Generation framework to generate responses, which are evaluated using both automated techniques and human evaluators on four specific metrics relevant to our application. We find that models vary significantly in their performance and that instruction tuned Indic models do not always perform well on Indic language queries. Further, we empirically show that factual correctness is generally lower for responses to Indic queries compared to English queries. Finally, our qualitative work shows that code-mixed and culturally relevant queries in our dataset pose challenges to evaluated models.
CRISS: A Retrieval-Augmented AI Chatbot for Assisting Cancer Registrars
University of Missouri（密苏里大学）；Missouri Cancer Registry and Research Center（密苏里癌症登记与研究中心）；MU Institute for Data Science and Informatics（密苏里大学数据科学与信息学研究所）
UniD$^3$: A Knowledge Graph-Enhanced RAG Framework for Drug-Disease Discovery and Reasoning
Department of Health Outcomes and Biomedical Informatics, University of Florida（佛罗里达大学健康成果与生物医学信息学系）；Department of Hematology, H. Lee Moffitt Cancer Center and Research Institute（血液科，H. Lee Moffitt癌症中心与研究院）；Center for Bioinformatics and Computational Biology, Houston Methodist Research Institute（生物信息学与计算生物学中心，休斯顿方法主义研究学院）；Department of Cardiothoracic Surgery, Weill Cornell Medicine, Cornell University（心胸外科，Weill Cornell医学，康奈尔大学）；Department of Biostatistics and Health Data Science, Indiana University School of Medicine（生物统计学与健康数据科学系，印第安纳大学医学院）
Adversarial Databases Improve Success in Retrieval-based Large Language Models
Keck Data Science Institute, Pepperdine University（佩珀代因大学凯克数据科学研究所）；David Geffen School of Medicine, University of California, Los Angeles（加州大学洛杉矶分校大卫·格芬医学院）
AgroLLM: Connecting Farmers and Agricultural Practices through Large Language Models for Enhanced Knowledge Transfer and Practical Application
Pittsburg State University（匹兹堡州立大学）；Oxford Brookes University（牛津布鲁克斯大学）；Kelce College of Business, Pittsburg State University（匹兹堡州立大学凯尔斯商学院）；MSK Cancer Center（纪念斯隆-凯特琳癌症中心）
ElectriQ: A Benchmark for Assessing the Response Capability of Large Language Models in Power Marketing
Systems Engineering Institute, Xi’an Jiaotong University（系统工程研究所，西安交通大学）；State Key Laboratory of Multiphase Flow in Power Engineering, School of Energy and Power Engineering, Xi’an Jiaotong University（电力工程多相流国家重点实验室，能源与动力工程学院，西安交通大学）；School of Electronic Science and Engineering, Xi'an Jiaotong University（电子科学与工程学院，西安交通大学）；Organizational Management Department, School of Management, Xi’an Jiaotong University（组织管理部，管理学院，西安交通大学）
