发表机构
The University of Tokyo; Xiaohongshu Inc.; Tsinghua University（东京大学; 小红书公司; 清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究构建了含4759个中文新词汇的基准CNeo-Bench，搭配两层评估框架评估18个大语言模型，发现模型在定义生成上表现差，存在识别-操作差距，少样本提示仍无法解决部分难题。
AI 中文摘要
中文新词汇运用了多样且独特的语言机制，例如其他语言中罕见的语音替代（如“886”对应“bye-bye”）和视觉字符分解。我们推出了CNeo-Bench，这是一个包含4759个此类新词汇的基准数据集，配有参考定义，根据每个表达背后的语言机制将其分为5个顶级类别和9个子类别。CNeo-Bench搭配了一个两层评估框架，该框架将模型能否描述新词汇与能否基于其底层机制进行操作区分开来。对18个大语言模型（LLM）的评估显示，中文新词汇仍是一个开放性挑战；大多数模型在定义生成上的表现低于40%，且在多个子类别中出现了系统性的识别-操作差距：模型能正确描述新词汇，但在源形式恢复任务中，会用语义等价物（同义词）替代源形式而非生成源形式本身。对1058个难题的少样本分析显示，上下文示例可解决许多困难案例，但仍存在相当一部分错误，表明仅靠提示无法解决的挑战依然存在。
英文摘要
Chinese neologisms exploit diverse and unique linguistic mechanisms, such as phonetic substitution (e.g., 886 for ``bye-bye'') and visual character decomposition that are rare in other languages. We introduce CNeo-Bench, a benchmark of 4,759 such neologisms with reference definitions, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression. CNeo-Bench is paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whether it can operate on its underlying mechanism. Evaluating 18 LLMs, we find that Chinese neologisms remain an open challenge; most models fall below 40\% on definition generation, and on several subcategories a systematic recognition-manipulation gap emerges: models describe neologisms correctly but, in source-form restoration tasks, substitute a semantic equivalent (paraphrase) for the source form rather than producing the source form itself. A few-shot analysis on 1,058 hard items shows that in-context examples can solve many difficult cases, but leave a noticeable portion of errors remaining, indicating challenges beyond prompting alone can address.
CommentsWork in progress
