聚合与个性化评判在商业创意评估中的对比：来自专家分歧的证据
Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement
AI总结：
本文研究了在商业创意评估中，自动评判应接近聚合共识还是个体评判的问题，通过PBIG-DATA数据集发现个性化评判更符合评估者，而聚合评判存在较大分歧。
摘要：
评估LLM生成的商业创意往往比生成它们更难扩展。与标准NLP基准不同，商业创意评估依赖多维标准如可行性、新颖性、差异化、用户需求和市场规模，专家评判常存在分歧。本文研究了这种分歧引发的方法论问题：自动评判应近似聚合共识还是模拟个体评判？我们引入了PBIG-DATA数据集，包含约3000个个体评分，来自六个商业导向维度：具体性、技术有效性、创新性、竞争优势、需求有效性及市场规模。分析显示，对细粒度顺序评分存在显著分歧，而粗粒度选择下一致性更高，表明结构异质性而非随机噪声。我们随后比较了三种评判配置：仅基于规则的零样本评判、基于混合评估者历史的聚合评判，以及基于目标评估者评分历史的个性化评判。在不同维度和模型大小下，个性化评判更贴近相应评估者，而评估者一致性仅在个性化条件下与评判生成的推理相似性相关。这些结果表明，在多元评估设置中，聚合标签可能成为脆弱目标，并促使针对评估者条件的评判设计。
英文摘要：
Evaluating LLM-generated business ideas is often harder to scale than generating them. Unlike standard NLP benchmarks, business idea evaluation relies on multi-dimensional criteria such as feasibility, novelty, differentiation, user need, and market size, and expert judgments often disagree. This paper studies a methodological question raised by such disagreement: should an automatic judge approximate an aggregate consensus, or model evaluators individually? We introduce PBIG-DATA, a dataset of approximately 3,000 individual scores across 300 patent-grounded product ideas, provided by domain experts on six business-oriented dimensions: specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Analyses show substantial expert disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection, suggesting structured heterogeneity rather than random noise. We then compare three judge configurations: a rubric-only zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator's scoring history. Across dimensions and model sizes, personalized judges align more closely with the corresponding evaluator than aggregate judges, and evaluator agreement correlates with similarity of judge-generated reasoning only under personalized conditioning. These results indicate that pooled labels can be a fragile target in pluralistic evaluation settings and motivate evaluator-conditioned judge designs for business idea assessment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
