arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.05035cs.CLcs.AI
收藏

情境化自然语言解释的场景与方法

Scenarios and Approaches for Situated Natural Language Explanations

Tohoku University（东北大学）；Vector Institute for Artificial Intelligence（人工智能矢量研究所）；Dalhousie University（达尔豪斯大学）；Stevens Institute of Technology（史蒂文斯理工学院）

Pengshuo Qiu, Frank Rudzicz, Zining Zhu

备注：
8 pages, 4 figures
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出情境化自然语言解释基准数据集 SBE，通过三类提示方法在多种预训练语言模型上定量评估 LLMs 对受众情境的适配能力，并揭示了人设提示与上下文学习提示的局限性。

摘要：

大语言模型（LLMs）可用于生成适应不同用户情境的自然语言解释（NLE）。然而，目前尚缺乏对这种适应程度的定量评估。为弥补这一空白，我们收集了一个基准数据集 Situation-Based Explanation。该数据集包含 100 个待解释项（explanandum）。每个待解释项都配有面向三种不同受众类型（如教育者、学生和专业人士）的解释，使我们能够评估这些解释在多大程度上满足不同群体（如学生、教师和家长）的特定信息需求与情境。对于每个“待解释项与受众配对”的情境，我们都包含一条人工撰写的解释。这些人工解释使我们能够计算量化 LLMs 将解释适配到具体情境程度的分数。在多种不同规模的预训练语言模型上，我们考察了三类提示方法：基于规则的提示、元提示（meta-prompting）和上下文学习提示（in-context learning prompting）。我们发现：1）语言模型能够生成使解释更精确对齐目标情境的提示；2）通过提示“你是一个乐于助人的助手……”来显式建模“助手”人设，对于情境化 NLE 任务并非必要的提示技术；3）上下文学习提示只能帮助 LLMs 学习示范模板，而无法提升其推理性能。SBE 及我们的分析为未来生成情境化自然语言解释的研究提供了支持。

英文摘要：

Large language models (LLMs) can be used to generate natural language explanations (NLE) that are adapted to different users' situations. However, there is yet to be a quantitative evaluation of the extent of such adaptation. To bridge this gap, we collect a benchmarking dataset, Situation-Based Explanation. This dataset contains 100 explanandums. Each explanandum is paired with explanations targeted at three distinct audience types-such as educators, students, and professionals-enabling us to assess how well the explanations meet the specific informational needs and contexts of these diverse groups e.g. students, teachers, and parents. For each "explanandum paired with an audience" situation, we include a human-written explanation. These allow us to compute scores that quantify how the LLMs adapt the explanations to the situations. On an array of pretrained language models with varying sizes, we examine three categories of prompting methods: rule-based prompting, meta-prompting, and in-context learning prompting. We find that 1) language models can generate prompts that result in explanations more precisely aligned with the target situations, 2) explicitly modeling an "assistant" persona by prompting "You are a helpful assistant..." is not a necessary prompt technique for situated NLE tasks, and 3) the in-context learning prompts only can help LLMs learn the demonstration template but can't improve their inference performance. SBE and our analysis facilitate future research towards generating situated natural language explanations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑