arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.13605cs.CL
收藏

KorNAT: 面向韩国社会价值观与常识的大语言模型对齐基准

KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge

KAIST AI(韩国科学技术院人工智能学院);DATUMO Inc.(DATUMO公司);Seoul National University Bundang Hospital(首尔大学盆唐医院);NAVER AI Lab(NAVER人工智能实验室)

Jiyoung Lee, Minwoo Kim, Seungho Kim, Junghwan Kim, Seunghyun Won, Hwaran Lee, Edward Choi

AI总结:

本研究提出国家对齐概念,构建首个面向韩国社会价值观与常识的LLM对齐基准KorNAT,经大规模调研与多轮审核形成万道选择题,实验显示多数现有模型仍有提升空间,且该基准已获韩国政府相关机构批准。

摘要:

大语言模型(LLM)若要在特定国家有效部署,必须具备对该国文化与基础知识的理解。为此,我们提出“国家对齐(National Alignment)”概念,从社会价值观对齐与常识对齐两个维度衡量大语言模型与目标国家的对齐程度:社会价值观对齐评估模型对该国特有社会价值观的理解程度,常识对齐则检验模型对该国相关基础知识的掌握程度。\n我们构建了KorNAT——首个衡量与韩国国家对齐程度的基准。其中社会价值观数据集的真值标签来自涵盖6174名韩国独立参与者的大规模调研;常识数据集的样本基于韩国教科书与GED(普通教育发展)参考资料构建。KorNAT分别包含4000道社会价值观类选择题与6000道常识类选择题,数据集构建流程经精心设计,以统计抽样理论为基础,并通过多轮人工审核完善。\n对7个大语言模型的实验结果显示,仅有少数模型达到我们设定的参考分数,表明仍有进一步提升空间。KorNAT已通过一家负责数据集质量评估的政府附属机构的审核,获得政府批准。数据集样本与详细评估协议可参见https://huggingface.co/datasets/jiyounglee0523/KorNAT 与 https://github.com/jiyounglee-0523/KorNAT。

英文摘要:

For Large Language Models (LLMs) to be effectively deployed in a specific country, they must possess an understanding of the nation's culture and basic knowledge. To this end, we introduce National Alignment, which measures an alignment between an LLM and a targeted country from two aspects: social value alignment and common knowledge alignment. Social value alignment evaluates how well the model understands nation-specific social values, while common knowledge alignment examines how well the model captures basic knowledge related to the nation. We constructed KorNAT, the first benchmark that measures national alignment with South Korea. For the social value dataset, we obtained ground truth labels from a large-scale survey involving 6,174 unique Korean participants. For the common knowledge dataset, we constructed samples based on Korean textbooks and GED reference materials. KorNAT contains 4K and 6K multiple-choice questions for social value and common knowledge, respectively. Our dataset creation process is meticulously designed and based on statistical sampling theory and was refined through multiple rounds of human review. The experiment results of seven LLMs reveal that only a few models met our reference score, indicating a potential for further enhancement. KorNAT has received government approval after passing an assessment conducted by a government-affiliated organization dedicated to evaluating dataset quality. Samples and detailed evaluation protocols of our dataset can be found in https://huggingface.co/datasets/jiyounglee0523/KorNAT and https://github.com/jiyounglee-0523/KorNAT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑