基础模型在后苏联世界中“烹饪”什么?
What are Foundation Models Cooking in the Post-Soviet World?
AI总结:
本研究构建多模态数据集BORSch,评估基础模型对后苏联菜肴起源的识别,发现模型存在语言偏差,且QA不足以衡量文化理解。
摘要:
后苏联国家的文化是复杂的,由动荡的历史塑造,这段历史持续影响着当前事件。在本研究中,我们通过构建BORSch数据集来调查基础模型对后苏联文化饮食知识的掌握情况,该数据集是一个多模态数据集,包含1147道俄语菜肴和823道乌克兰语菜肴,聚焦于后苏联地区。我们证明,领先的模型在纯文本和多模态问答(QA)中均难以正确识别后苏联国家菜肴的起源,反而过度预测与提问语言相关的国家。通过对预训练数据的分析,我们表明这些结果可以通过误导性的菜肴-起源共现以及俄乌代码混合等语言现象来解释。最后,为了超越基于QA的评估,我们测试了模型生成菜肴准确视觉描述的能力。该任务与QA之间的弱相关性表明,仅靠QA可能不足以评估文化理解。为了促进进一步研究,我们将在https://github.com/alavrouk/BORSch公开提供BORSch。
英文摘要:
The culture of the Post-Soviet states is complex, shaped by a turbulent history that continues to influence current events. In this study, we investigate the Post-Soviet cultural food knowledge of foundation models by constructing BORSch, a multimodal dataset encompassing 1147 and 823 dishes in the Russian and Ukrainian languages, centered around the Post-Soviet region. We demonstrate that leading models struggle to correctly identify the origins of dishes from Post-Soviet nations in both text-only and multimodal Question Answering (QA), instead over-predicting countries linked to the language the question is asked in. Through analysis of pretraining data, we show that these results can be explained by misleading dish-origin co-occurrences, along with linguistic phenomena such as Russian-Ukrainian code mixing. Finally, to move beyond QA-based assessments, we test models' abilities to produce accurate visual descriptions of dishes. The weak correlation between this task and QA suggests that QA alone may be insufficient as an evaluation of cultural understanding. To foster further research, we will make BORSch publicly available at https://github.com/alavrouk/BORSch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。