DLAMA：一个用于策划文化多样性事实以探测预训练语言模型知识的框架
DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Models
AI总结：
针对现有多语言事实探测基准mLAMA存在西方文化偏差的问题，提出DLAMA框架构建含78259个三元组的跨文化均衡基准DLAMA-v1，验证了预训练模型的事实知识存在文化偏向性。
摘要：
目前已有少量基准数据集发布，用于评估预训练语言模型的事实性知识。这些基准（如LAMA、ParaRel）主要以英语开发，后续经翻译形成新的多语言版本（如mLAMA、mParaRel）。这些多语言基准上的结果表明，使用英语提示从多语言模型中召回事实的表现，通常显著优于且比使用非英语提示更稳定。我们的分析显示，mLAMA存在偏向西方国家事实的偏差，这可能影响模型探测的公平性。我们提出一个新框架，用于从Wikidata（维基数据）中策划具备文化多样性的事实三元组。基于该框架构建了新基准DLAMA-v1，它包含来自三组对比文化的事实三元组，涵盖20个关系谓词，总计78259个三元组。这三组对比文化分别代表（阿拉伯与西方）、（亚洲与西方）、（南美与西方）国家的事实。借助更均衡的基准DLAMA-v1我们证实：mBERT在西方事实上的表现优于非西方事实，而单语阿拉伯语、英语、韩语模型在与其文化相近的事实上表现更优。此外，无论是单语还是多语言模型，即便预测错误，其给出的预测结果也往往与正确标签存在文化或地理层面的关联。
英文摘要：
A few benchmarking datasets have been released to evaluate the factual knowledge of pretrained language models. These benchmarks (e.g., LAMA, and ParaRel) are mainly developed in English and later are translated to form new multilingual versions (e.g., mLAMA, and mParaRel). Results on these multilingual benchmarks suggest that using English prompts to recall the facts from multilingual models usually yields significantly better and more consistent performance than using non-English prompts. Our analysis shows that mLAMA is biased toward facts from Western countries, which might affect the fairness of probing models. We propose a new framework for curating factual triples from Wikidata that are culturally diverse. A new benchmark DLAMA-v1 is built of factual triples from three pairs of contrasting cultures having a total of 78,259 triples from 20 relation predicates. The three pairs comprise facts representing the (Arab and Western), (Asian and Western), and (South American and Western) countries respectively. Having a more balanced benchmark (DLAMA-v1) supports that mBERT performs better on Western facts than non-Western ones, while monolingual Arabic, English, and Korean models tend to perform better on their culturally proximate facts. Moreover, both monolingual and multilingual models tend to make a prediction that is culturally or geographically relevant to the correct label, even if the prediction is wrong.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
