QSTN: 一种用于大型语言模型鲁棒问卷推断的模块化框架
QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models
AI总结:
QSTN是一种模块化框架,通过无需编码的界面实现大型语言模型在问卷推断中的稳健评估与高效响应生成。
摘要:
我们介绍了QSTN,一个开源的Python框架,用于系统地从问卷式提示生成响应,以支持基于大型语言模型(LLMs)的计算机模拟调查和标注任务。QSTN能够实现问卷呈现、提示扰动和响应生成方法的稳健评估。我们的广泛评估(>4000万份调查响应)显示,问题结构和响应生成方法对生成的调查响应与人类答案的一致性有显著影响。我们还发现,通过改变呈现方法,可以以较少的计算成本获得答案。此外,我们提供了一个无需编码的用户界面,使研究人员能够无需编码知识设置稳健的LLM实验。我们希望QSTN将支持未来基于LLM的研究的可重复性和可靠性。
英文摘要:
We introduce QSTN, an open-source Python framework for systematically generating responses from questionnaire-style prompts to support in-silico surveys and annotation tasks with large language models (LLMs). QSTN enables robust evaluation of questionnaire presentation, prompt perturbations, and response generation methods. Our extensive evaluation (>40 million survey responses) shows that question structure and response generation methods have a significant impact on the alignment of generated survey responses with human answers. We also find that answers can be obtained for a fraction of the compute cost, by changing the presentation method. In addition, we offer a no-code user interface that allows researchers to set up robust experiments with LLMs \emph{without coding knowledge}. We hope that QSTN will support the reproducibility and reliability of LLM-based research in the future.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。