可视对话框的MRR和NDCG型号的扩展
Ensemble of MRR and NDCG models for Visual Dialog
摘要：
评估一个可以用人类语言交谈并理解视觉内容的人工智能智能体是一项挑战。生成度量，例如BLEU分数，更倾向于正确的语法而不是语义。因此，经常使用判别方法，其中代理对一组候选选项进行排名。平均倒数秩（MRR）度量通过考虑单个人类来源答案的秩来评估模型性能。然而，这种方法提出了一个新的挑战：答案的模糊性和同义性，例如，语义等同（例如，“是”和“是”）。为了解决这个问题，归一化的折扣累积增益（NDCG）度量已被用于通过密集注释捕获所有正确答案的相关性。然而，NDCG指标倾向于通常适用的不确定答案，如“我不知道”。制作一个在MRR和NDCG指标上都表现出色的模型是具有挑战性的。理想情况下，人工智能智能体应该回答类似人类的回复，并验证任何答案的正确性。为了解决这个问题，我们描述了一个两步的非参数排名方法，可以合并强大的MRR和NDCG模型。使用我们的方法，我们设法保持大多数MRR最先进的性能（70.41%对71.24%）和NDCG最先进的性能（72.16%对75.35%）。此外，我们的方法赢得了最近的Visual Dialog 2020挑战赛。源代码可在https://github.com/idansc/mrr-ndcg上获得。
英文摘要：
Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) metric evaluates the model performance by taking into account the rank of a single human-derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (e.g., `yeah' and `yes'). To address this, the normalized discounted cumulative gain (NDCG) metric has been used to capture the relevance of all the correct answers via dense annotations. However, the NDCG metric favors the usually applicable uncertain answers such as `I don't know. Crafting a model that excels on both MRR and NDCG metrics is challenging. Ideally, an AI agent should answer a human-like reply and validate the correctness of any answer. To address this issue, we describe a two-step non-parametric ranking approach that can merge strong MRR and NDCG models. Using our approach, we manage to keep most MRR state-of-the-art performance (70.41% vs. 71.24%) and the NDCG state-of-the-art performance (72.16% vs. 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at https://github.com/idansc/mrr-ndcg.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
