arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20135cs.CLcs.AIcs.IR
收藏

KNIGHT: 基于知识图谱的多选题生成与自适应难度校准

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

University of Tehran(塔里班大学);Independent Researcher(独立研究员);Amirkabir University of Technology(阿米尔卡比尔技术大学);TEIAS Institute(TEIAS研究所)

Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari, Mahdi Noori, Farhan Farsi, Behnam Bahrak

AI总结:

KNIGHT通过构建领域特定知识图谱,实现高效生成多选题数据集,支持自适应难度控制,提升生成效率与质量,验证了其在多个领域内的有效性。

摘要:

随着大语言模型(LLMs)的兴起,它们在检索增强生成(RAG)等应用中变得至关重要。然而,评估这些系统仍受制于构建专用评估数据集的时间和成本。我们介绍了KNIGHT,一种基于LLM的知识图谱驱动框架,用于从外部来源生成多选题(MCQ)数据集。KNIGHT构建了一个主题特定的知识图谱,这是一个结构化且简洁的实体和关系摘要,可以重复使用以生成由教师控制的难度级别,包括多跳问题,而无需反复重新输入完整源文本。该知识图谱充当一个压缩、可重用的状态,使问题生成成为对图的廉价读取。我们将在维基百科/Wikidata上实例化KNIGHT,同时保持框架的领域和本体无关性。作为案例研究,KNIGHT在历史、生物学和数学领域生成了六个MCQ数据集。我们评估了五个标准:流畅性、无歧义性(单个正确答案)、主题相关性、选项唯一性和给定源提供的答案性(作为幻觉的代理)。结果表明,KNIGHT能够通过可重用的图表示实现令牌和成本高效的生成,实现了这些标准的高质量,且模型排名与MMLU风格基准一致,同时支持主题特定和难度控制的评估。

英文摘要:

With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured and parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This knowledge graph acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable graph representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑