Natural Language to Code Generation has made significant progress in recent years with the advent of Large Language Models(LLMs). While generation for general-purpose languages like C, C++, and Python has improved significantly, LLMs struggle with custom function names in Domain Specific Languages or DSLs. This leads to higher hallucination rates and syntax errors, specially for DSLs having a high number of custom function names. Additionally, constant updates to function names add to the challenge as LLMs need to stay up-to-date. In this paper, we present optimizations for using Retrieval Augmented Generation (or RAG) with LLMs for DSL generation along with an ablation study comparing these strategies. We generated a train as well as test dataset with a DSL to represent automation tasks across roughly 700 APIs in public domain. We used the training dataset to fine-tune a Codex model for this DSL. Our results showed that the fine-tuned model scored the best on code similarity metric. With our RAG optimizations, we achieved parity for similarity metric. The compilation rate, however, showed that both the models still got the syntax wrong many times, with RAG-based method being 2 pts better. Conversely, hallucination rate for RAG model lagged by 1 pt for API names and by 2 pts for API parameter keys. We conclude that an optimized RAG model can match the quality of fine-tuned models and offer advantages for new, unseen APIs.
Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study
Miner School of Computer and Information Sciences, UMass Lowell, MA, USA(UMass Lowell 计算机与信息科学学院);Manning College of Information and Computer Sciences, UMass Amherst, MA, USA(UMass Amherst 信息与计算机科学学院);Center for Healthcare Organization and Implementation Research, VA Bedford Health Care, MA, USA(VA Bedford医疗中心医疗组织与实施研究中心)
A First Context-Free Grammar Applied to Nawatl Corpora Augmentation
Laboratoire Informatique d’Avignon(阿维尼翁信息实验室);Avignon Université(阿维尼昂大学);Facultad de Matemáticas(数学系);Instituto de Investigaciones en Educación(教育研究所);Universidad Veracruzana(韦拉克鲁斯大学)
Knowledge Graph-Enhanced Zero-Shot Topic Classification: A Multi-Strategy Comparative Study
A2I Lab, School of Computing, Wichita State University(A2I实验室,计算学院,威斯康星州立大学);EIS Lab, College of Engineering, University of Nevada, Reno(EIS实验室,工程学院,内华达大学,里诺)
Evaluating Large Language Models on Quantum Mechanics: A Comparative Study Across Diverse Models and Tasks
Quantum Science and Technology Laboratory, Physical Research Laboratory, Navrangpura, Ahmedabad 380009, India(量子科学与技术实验室,物理研究实验室,Ahmedabad);Indian Institute of Technology Gandhinagar, Palaj, Gandhinagar 382055, India(印度理工学院冈达塞瓦尔)
Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework
Department of Electrical Engineering;Computer Science University of Toledo Toledo, USA;Institute of Mathematical Sciences Claremont Graduate University Claremont, USA;Department of Bioengineering University of Toledo Toledo, USA;Department of Computer Science Bowling Green State University Bowling Green, USA
SIGMA: Refining Large Language Model Reasoning via Sibling-Guided Monte Carlo Augmentation
Beihang University(北京航空航天大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Leicester(莱斯特大学);Nanyang Technological University(南洋理工大学)
From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection
School of Computer and Computing Science, Hangzhou City University(杭州城市大学计算机与计算科学学院);College of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院)