模拟意义，永不再来！引入ICR：一种用于评估LLM文本摘要中意义的象征-解释学度量标准
Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summaries
AI总结：
本文提出ICR度量标准，用于评估LLM生成文本摘要中的意义，通过归纳内容分析和反思主题分析，超越词法相似性度量，评估语义准确性和意义一致性。
摘要：
人类语言中的意义是关系性的、依赖于语境的，并且是涌现的，源于符号动态系统的相互作用，而不是固定的词-概念映射。在计算环境中，这种象征性和解释性复杂性使得意义的生成和评估变得更加困难。本文提出一个跨学科框架，用于研究大语言模型（LLM）生成语言中的意义，通过将象征学和解释学与定性研究方法相结合。我们回顾了关于意义与机器的先前研究，考察了语言符号如何在静态和上下文化的嵌入模型中转化为向量表示，并识别了统计近似与人类解释性意义之间的差距。然后我们引入了归纳概念评分（ICR）度量标准，这是一种基于归纳内容分析和反思主题分析的定性评估方法，旨在评估LLM输出中的语义准确性和意义一致性，而不仅仅是词法相似性度量。我们将在五个数据集（N=50至800）上对LLM生成和人工生成的主题摘要进行实证比较。尽管LLM在语言相似性上表现优异，但在语义准确性上却表现欠佳，特别是在捕捉基于语境的意义方面。随着数据集规模的增大，性能有所提升，但不同模型之间仍存在差异，这可能反映了重复概念和意义的频率和连贯性方面的差异。我们最后得出结论，应倡导利用系统性定性解释实践来评估LLM生成输出中从参考文本中意义的评估框架。
英文摘要：
Meaning in human language is relational, context dependent, and emergent, arising from dynamic systems of signs rather than fixed word-concept mappings. In computational settings, this semiotic and interpretive complexity complicates the generation and evaluation of meaning. This article proposes an interdisciplinary framework for studying meaning in large language model (LLM) generated language by integrating semiotics and hermeneutics with qualitative research methods. We review prior scholarship on meaning and machines, examining how linguistic signs are transformed into vectorized representations in static and contextualized embedding models, and identify gaps between statistical approximation and human interpretive meaning. We then introduce the Inductive Conceptual Rating (ICR) metric, a qualitative evaluation approach grounded in inductive content analysis and reflexive thematic analysis, designed to assess semantic accuracy and meaning alignment in LLM-outputs beyond lexical similarity metrics. We apply ICR in an empirical comparison of LLM generated and human generated thematic summaries across five datasets (N = 50 to 800). While LLMs achieve high linguistic similarity, they underperform on semantic accuracy, particularly in capturing contextually grounded meanings. Performance improves with larger datasets but remains variable across models, potentially reflecting differences in the frequency and coherence of recurring concepts and meanings. We conclude by arguing for evaluation frameworks that leverage systematic qualitative interpretation practices when assessing meaning in LLM-generated outputs from reference texts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
