XATU：一个面向可解释文本更新的细粒度指令基准
XATU: A Fine-grained Instruction-based Benchmark for Explainable Text Updates
AI总结：
本文提出XATU，首个细粒度指令的可解释文本编辑基准，涵盖多难度任务，结合LLM与人工标注，验证了指令调优和解释在提升LLM编辑能力中的关键作用。
摘要：
文本编辑是一项关键任务，旨在修改文本以更好地符合用户意图。然而，现有的文本编辑基准数据集仅包含粗粒度指令，且缺乏可解释性，导致输出与黄金参考中预期的更改存在偏差。为了全面研究大型语言模型（LLMs）的文本编辑能力，本文引入了XATU，这是第一个专门为基于细粒度指令的可解释文本编辑设计的基准。XATU考虑了不同难度（简化、语法检查、事实核查等）的更细粒度文本编辑任务，涵盖了词汇、句法、语义和知识密集型编辑方面。为了增强可解释性，我们结合了基于LLM的标注和人工标注，形成了一个包含细粒度指令和黄金标准编辑解释的基准。通过针对我们的基准评估现有LLMs，我们展示了指令调优的有效性以及底层架构对各种编辑任务的影响。此外，大量实验揭示了在文本编辑任务中，解释在微调语言模型中的重要作用。该基准将开源以支持复现并促进未来研究，网址为\url{https://github.com/megagonlabs/xatu}。
英文摘要：
Text editing is a crucial task of modifying text to better align with user intents. However, existing text editing benchmark datasets contain only coarse-grained instructions and lack explainability, thus resulting in outputs that deviate from the intended changes outlined in the gold reference. To comprehensively investigate the text editing capabilities of large language models (LLMs), this paper introduces XATU, the first benchmark specifically designed for fine-grained instruction-based explainable text editing. XATU considers finer-grained text editing tasks of varying difficulty (simplification, grammar check, fact-check, etc.), incorporating lexical, syntactic, semantic, and knowledge-intensive edit aspects. To enhance interpretability, we combine LLM-based annotation and human annotation, resulting in a benchmark that includes fine-grained instructions and gold-standard edit explanations. By evaluating existing LLMs against our benchmark, we demonstrate the effectiveness of instruction tuning and the impact of underlying architecture across various editing tasks. Furthermore, extensive experimentation reveals the significant role of explanations in fine-tuning language models for text editing tasks. The benchmark will be open-sourced to support reproduction and facilitate future research at~\url{https://github.com/megagonlabs/xatu}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
