评估大型语言模型生成合成数据的多维度评估框架
A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models
AI总结：
本文提出开源框架 SynEval，从保真度、效用和隐私保护三方面量化评估 LLM 生成的表格合成数据，并在 ChatGPT、Claude 和 Llama 的产品评论数据上验证其指标权衡分析能力。
摘要：
生成式人工智能与大型语言模型（LLM）的快速发展为生产合成数据开辟了新途径，尤其是在产品评论等结构化表格格式领域。尽管存在潜在收益，隐私泄露方面的担忧也随之出现，特别是当个人信息被用于训练数据集时。此外，目前缺乏一个能够定量衡量所生成合成数据质量及其对下游任务效用的综合评估框架。为应对这一空白，我们提出 SynEval，这是一个开源评估框架，旨在通过一组多样化的评估指标，评估合成生成的表格数据的保真度、效用和隐私保护能力。我们将所提出的框架 SynEval 应用于由三个最先进 LLM——ChatGPT、Claude 和 Llama——生成的合成产品评论数据，验证了其有效性。实验结果阐明了合成数据生成情境中各类评估指标之间的权衡关系。此外，SynEval 是研究人员和从业者处理合成表格数据时的关键工具，使其能够审慎判断所生成数据对其特定应用的适用性，并重点维护用户隐私。
英文摘要：
The rapid advancements in generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data, particularly in the realm of structured tabular formats, such as product reviews. Despite the potential benefits, concerns regarding privacy leakage have surfaced, especially when personal information is utilized in the training datasets. In addition, there is an absence of a comprehensive evaluation framework capable of quantitatively measuring the quality of the generated synthetic data and their utility for downstream tasks. In response to this gap, we introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data via a suite of diverse evaluation metrics. We validate the efficacy of our proposed framework - SynEval - by applying it to synthetic product review data generated by three state-of-the-art LLMs: ChatGPT, Claude, and Llama. Our experimental findings illuminate the trade-offs between various evaluation metrics in the context of synthetic data generation. Furthermore, SynEval stands as a critical instrument for researchers and practitioners engaged with synthetic tabular data,, empowering them to judiciously determine the suitability of the generated data for their specific applications, with an emphasis on upholding user privacy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
