推进学术聊天机器人：非传统输出的评估
Advancing Academic Chatbots: Evaluation of Non Traditional Outputs
AI总结：
本研究评估了非传统学术输出生成，比较了两种检索策略并测试了LLMs在幻灯片和播客脚本生成中的表现。
摘要：
大多数大型语言模型的评估都集中在事实性问答或短摘要等标准任务上。本研究在两个方向上扩展了这一范围：首先，通过比较两种检索策略，基于知识图的Graph RAG和混合关键词语义搜索的Advanced RAG，用于问答任务；其次，评估LLMs是否能生成高质量的非传统学术输出，特别是幻灯片和播客脚本。我们实现了一个结合Meta的LLaMA 3 70B开源权重和OpenAI的GPT 4o mini API的原型。问答性能通过人类评分在十一项质量维度上的评估以及大规模交叉验证的大语言模型法官进行评估。GPT 4o mini与Advanced RAG产生了最准确的响应。Graph RAG提供了有限的改进，导致更多的幻觉，部分原因是其结构复杂性和手动设置。幻灯片和播客生成通过文档基础检索测试。GPT 4o mini再次表现最佳，尽管LLaMA 3在叙述连贯性上显示出潜力。人类评审员对于检测布局和风格缺陷至关重要，突显了在评估新兴学术输出时结合人类和LLM评估的必要性。
英文摘要：
Most evaluations of large language models focus on standard tasks such as factual question answering or short summarization. This research expands that scope in two directions: first, by comparing two retrieval strategies, Graph RAG, structured knowledge-graph based, and Advanced RAG, hybrid keyword-semantic search, for QA; and second, by evaluating whether LLMs can generate high quality non-traditional academic outputs, specifically slide decks and podcast scripts. We implemented a prototype combining Meta's LLaMA 3 70B open weight and OpenAI's GPT 4o mini API based. QA performance was evaluated using both human ratings across eleven quality dimensions and large language model judges for scalable cross validation. GPT 4o mini with Advanced RAG produced the most accurate responses. Graph RAG offered limited improvements and led to more hallucinations, partly due to its structural complexity and manual setup. Slide and podcast generation was tested with document grounded retrieval. GPT 4o mini again performed best, though LLaMA 3 showed promise in narrative coherence. Human reviewers were crucial for detecting layout and stylistic flaws, highlighting the need for combined human LLM evaluation in assessing emerging academic outputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
