KemenkeuGPT：利用大语言模型处理印度尼西亚政府金融数据与法规以增强决策能力
KemenkeuGPT: Leveraging a Large Language Model on Indonesia's Government Financial Data and Regulations to Enhance Decision Making
AI总结：
本研究开发了基于LangChain、RAG和微调的KemenkeuGPT模型，用于处理印度尼西亚政府金融数据与法规，将准确率从35%提升至61%，并验证了其辅助决策的潜力。
摘要：
数据对于循证决策和改善公共服务至关重要，包括印度尼西亚共和国财政部的公共服务。然而，政府金融数据和法规的复杂性与动态性可能阻碍决策。本研究探讨了大语言模型（LLMs）应对这些挑战的潜力，重点关注印度尼西亚的金融数据和法规。尽管LLMs在金融领域表现出有效性，但其在印度尼西亚公共部门的应用尚未被探索。本研究采用迭代过程，使用LangChain结合检索增强生成（RAG）、提示工程和微调来开发KemenkeuGPT。2003年至2023年的数据集收集自印度尼西亚财政部、印度尼西亚统计局和国际货币基金组织（IMF）。对财政部官员的调查和访谈为模型提供了信息、增强了模型并对其进行了微调。我们使用人类反馈、基于LLM的评估和基准测试对模型进行了评估。模型的准确率从35%提高到61%，正确率从48%提高到64%。检索增强生成评估（RAGAS）框架显示，KemenkeuGPT实现了44%的正确率，以及73%的忠实度、40%的精确率和60%的召回率，优于其他几个基础模型。一位来自财政部的专家访谈表明，KemenkeuGPT有潜力成为决策的重要工具。这些结果有望通过持续的人类反馈得到进一步改善。
英文摘要：
Data is crucial for evidence-based policymaking and enhancing public services, including those at the Ministry of Finance of the Republic of Indonesia. However, the complexity and dynamic nature of governmental financial data and regulations can hinder decision-making. This study investigates the potential of Large Language Models (LLMs) to address these challenges, focusing on Indonesia's financial data and regulations. While LLMs are effective in the financial sector, their use in the public sector in Indonesia is unexplored. This study undertakes an iterative process to develop KemenkeuGPT using the LangChain with Retrieval-Augmented Generation (RAG), prompt engineering and fine-tuning. The dataset from 2003 to 2023 was collected from the Ministry of Finance, Statistics Indonesia and the International Monetary Fund (IMF). Surveys and interviews with Ministry officials informed, enhanced and fine-tuned the model. We evaluated the model using human feedback, LLM-based evaluation and benchmarking. The model's accuracy improved from 35% to 61%, with correctness increasing from 48% to 64%. The Retrieval-Augmented Generation Assessment (RAGAS) framework showed that KemenkeuGPT achieved 44% correctness with 73% faithfulness, 40% precision and 60% recall, outperforming several other base models. An interview with an expert from the Ministry of Finance indicated that KemenkeuGPT has the potential to become an essential tool for decision-making. These results are expected to improve with continuous human feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
