评估自动生成缝纫说明中的时空一致性
Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions
AI总结：
针对LLM生成的逐步缝纫说明，本文提出基于树的自动时空一致性评估指标，证明其更贴合人工错误计数与质量评分，并对文本相似度反事实样本更稳健。
摘要：
本文提出一种新颖的、基于树的自动评估指标，用于评估LLM生成的逐步装配说明；相比BLEU和BERT相似度分数等传统指标，该指标能更准确地反映构建过程的时空方面。我们将所提出的指标应用于缝纫说明领域，并表明该指标与人工标注的错误计数以及人类质量评分具有更好的相关性，证明了其在评估缝纫说明时空合理性方面的优越性。进一步实验表明，对于专门构造、用以干扰依赖文本相似度的指标的人工反事实样本，该指标比传统方法更稳健。
英文摘要：
In this paper, we propose a novel, automatic tree-based evaluation metric for LLM-generated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric's superiority for evaluating the spatiotemporal soundness of sewing instructions. Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
