透明参考-free 自动评估开放式用户调查回应
Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses
AI总结：
本文提出一种两阶段评估框架，用于评估人类开放式调查回应，通过去除无意义回应和评估努力、相关性和完整性，提升自动评估效果。
摘要：
开放式调查回应为营销研究提供有价值见解，但低质量回应不仅使研究人员负担手动筛选，还可能引起误导性结论，凸显了有效评估的必要性。现有自动评估方法针对LLM生成文本，无法充分评估具有独特特征的人类回应。为解决此类特征，我们提出一种专门针对人类调查回应的两阶段评估框架。首先，垃圾语过滤去除无意义回应。然后，通过LLM能力评估三个维度——努力、相关性和完整性，基于实际调查数据的实证分析。在英语和韩语数据集上的验证表明，我们的框架不仅优于现有指标，而且在实际应用如回应质量预测和回应拒绝中显示出高实用性，与专家评估显示强相关性。
英文摘要：
Open-ended survey responses provide valuable insights in marketing research, but low-quality responses not only burden researchers with manual filtering but also risk leading to misleading conclusions, underscoring the need for effective evaluation. Existing automatic evaluation methods target LLM-generated text and inadequately assess human-written responses with their distinct characteristics. To address such characteristics, we propose a two-stage evaluation framework specifically designed for human survey responses. First, gibberish filtering removes nonsensical responses. Then, three dimensions-effort, relevance, and completeness-are evaluated using LLM capabilities, grounded in empirical analysis of real-world survey data. Validation on English and Korean datasets shows that our framework not only outperforms existing metrics but also demonstrates high practical applicability for real-world applications such as response quality prediction and response rejection, showing strong correlations with expert assessment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
