arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.01708cs.CLcs.SI
收藏

考察关系对齐在大型语言模型中的作用

Examining the Role of Relationship Alignment in Large Language Models

Meta;Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学)

Kristen M. Altenburger, Hongda Jiang, Robert E. Kraut, Yi-Chia Wang, Jane Dwivedi-Yu

AI总结:

本文基于Facebook公开帖评数据评估Llama 3.0(70B)的关系对齐能力,发现其可仅凭原帖感知社会语境,但显式提供关系信息反而降低生成评论与人类评论的相似性。

摘要:

生成式AI在社交场景中的快速发展与部署,引出了一个重要问题:如何在保持准确性和真实性的同时,最佳地为用户实现个性化。本研究基于一个Facebook公开帖子—评论数据集,评估Llama 3.0(70B)在评论者与发帖者的性别、年龄和亲密度的不同组合下预测语义语气的能力,以及在LLM生成的评论中复现这些差异的能力。研究由两部分组成:第一部分评估不同社会关系类别之间语义语气的差异;第二部分以公开Facebook帖子为输入,考察Llama 3.0(70B)生成的评论与第一部分中人类评论之间的相似性。第一部分结果表明,纳入社会关系信息可提升模型预测人类评论语义语气的能力。然而,第二部分结果显示,即使提示中不包含社会情境信息,LLM生成的评论与人类评论对社会情境同样敏感,这表明LLM能够仅凭原帖理解语义。当我们在提示中纳入全部社会关系信息时,人类评论与LLM生成评论之间的相似性反而下降。这种不一致可能源于LLM的训练数据中并未包含社会情境信息。综合来看,这些结果证明LLM有能力从原帖中理解语义并作出与人类评论相似的回应,但也凸显了其仅依靠提示来泛化个性化评论的局限性。

英文摘要:

The rapid development and deployment of Generative AI in social settings raise important questions about how to optimally personalize them for users while maintaining accuracy and realism. Based on a Facebook public post-comment dataset, this study evaluates the ability of Llama 3.0 (70B) to predict the semantic tones across different combinations of a commenter's and poster's gender, age, and friendship closeness and to replicate these differences in LLM-generated comments. The study consists of two parts: Part I assesses differences in semantic tones across social relationship categories, and Part II examines the similarity between comments generated by Llama 3.0 (70B) and human comments from Part I given public Facebook posts as input. Part I results show that including social relationship information improves the ability of a model to predict the semantic tone of human comments. However, Part II results show that even without including social context information in the prompt, LLM-generated comments and human comments are equally sensitive to social context, suggesting that LLMs can comprehend semantics from the original post alone. When we include all social relationship information in the prompt, the similarity between human comments and LLM-generated comments decreases. This inconsistency may occur because LLMs did not include social context information as part of their training data. Together these results demonstrate the ability of LLMs to comprehend semantics from the original post and respond similarly to human comments, but also highlights their limitations in generalizing personalized comments through prompting alone.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑