arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.22971cs.CLcs.HC
收藏

AI 能猜出你知道什么吗?从通信日志中估计人类领域知识的大型语言模型性能比较

Can AI Guess What You Know? Performance Comparison of Large Language Models for Human Domain Knowledge Estimation From Communication Logs

Osaka Metropolitan University(大阪 metropolitan 大学)

Ko Watanabe, Shoya Ishimaru

AI总结:

本研究通过分析 Slack 日志,比较了七种大型语言模型在零样本设置下估计用户领域知识的能力,发现 Gemini 2.5 Flash 误差最低(MAE 21.13%),且估计精度与消息量弱相关。

摘要:

员工常常难以识别“谁知道什么”,导致组织生产力损失。我们研究大型语言模型(LLMs)是否能够直接从长期 Slack 日志中推断个人领域知识。通过分析来自 43 名用户的 27,188 条消息,我们评估了七个模型(包括 Gemini、Claude 和 GPT 系列),将其零样本估计与 27 名参与者的自我报告技能评分进行比较。Gemini 2.5 Flash 实现了最低误差(MAE 21.13%),而 GPT 模型显示出显著更大的差异。值得注意的是,估计精度仅弱依赖于消息量,表明更多的文本本身并不能保证更好的推断。这些发现证明了自动专业知识映射的可行性和当前局限性,强调了需要保护隐私的部署以及更丰富、结构感知的人类知识表示。

英文摘要:

Employees often struggle to identify ``who knows what,'' leading to organizational productivity losses. We investigate whether Large Language Models (LLMs) can infer individual domain knowledge directly from long-term Slack logs. Analyzing 27,188 messages from 43 users, we evaluated seven models (including Gemini, Claude, and GPT families) by comparing their zero-shot estimates against self-reported skill ratings from 27 participants. Gemini 2.5 Flash achieved the lowest error (MAE 21.13%), while GPT models showed significantly larger discrepancies. Notably, estimation accuracy depended only weakly on message volume, indicating that more text alone does not guarantee better inference. These findings demonstrate the feasibility and current limits of automated expertise mapping, highlighting the need for privacy-preserving deployments and richer, structure-aware representations of human knowledge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑