LLM认知领域基准测试：来自台湾客家文化的启示
Benchmarking Cognitive Domains for LLMs: Insights from Taiwanese Hakka Culture
AI总结：
本研究基于Bloom分类法构建了评估LLMs客家文化知识的六维度认知基准，并验证了RAG技术在提升事实检索准确性方面的有效性与在创造性任务中的局限性。
摘要：
本研究引入了一个综合基准，旨在评估大型语言模型（LLMs）在理解和处理文化知识方面的性能，并以客家文化为案例研究。基于Bloom分类法，本研究开发了一个多维度框架，系统地评估LLMs在六个认知领域的能力：记忆、理解、应用、分析、评估和创造。该基准超越了传统的单维度评估，通过提供对LLMs处理文化特定内容能力的更深入分析，从基础事实回忆到创意综合等高阶认知任务。此外，研究集成了检索增强生成（RAG）技术，以应对LLMs中少数群体文化知识表示的挑战，展示了RAG如何通过动态整合相关外部信息来提升模型性能。结果突显了RAG在提高所有认知领域准确性方面的有效性，特别是在需要精确检索和应用文化知识的任务中。然而，研究结果也揭示了RAG在创造性任务中的局限性，强调了进一步优化的必要性。该基准为在文化多元背景下评估和比较LLMs提供了稳健工具，为AI驱动的文化知识保存与传播的未来研究与发展提供了宝贵见解。
英文摘要：
This study introduces a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in understanding and processing cultural knowledge, with a specific focus on Hakka culture as a case study. Leveraging Bloom's Taxonomy, the study develops a multi-dimensional framework that systematically assesses LLMs across six cognitive domains: Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. This benchmark extends beyond traditional single-dimensional evaluations by providing a deeper analysis of LLMs' abilities to handle culturally specific content, ranging from basic recall of facts to higher-order cognitive tasks such as creative synthesis. Additionally, the study integrates Retrieval-Augmented Generation (RAG) technology to address the challenges of minority cultural knowledge representation in LLMs, demonstrating how RAG enhances the models' performance by dynamically incorporating relevant external information. The results highlight the effectiveness of RAG in improving accuracy across all cognitive domains, particularly in tasks requiring precise retrieval and application of cultural knowledge. However, the findings also reveal the limitations of RAG in creative tasks, underscoring the need for further optimization. This benchmark provides a robust tool for evaluating and comparing LLMs in culturally diverse contexts, offering valuable insights for future research and development in AI-driven cultural knowledge preservation and dissemination.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
