arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11784cs.CLcs.AI
收藏

MDCR:一个用于多文档条件推理的数据集

MDCR: A Dataset for Multi-Document Conditional Reasoning

MIT(麻省理工学院);AWS AI Labs(亚马逊云科技人工智能实验室)

Peter Baile Chen, Yi Zhang, Chunwei Liu, Sejal Gupta, Yoon Kim, Michael Cafarella

AI总结:

本文提出多文档条件推理数据集 MDCR,要求模型跨文档组合未知条件并求解最优结果,实验显示最新 LLM 在此复杂推理任务上仍存在明显局限。

摘要:

向不同个体提出的相同现实生活问题,可能会因其各自独特的情况而产生不同答案。例如,一名学生是否有资格获得奖学金,取决于资格条件,如所要求的专业或学位。ConditionalQA 被提出用于评估模型阅读文档并回答资格类问题的能力,同时考虑未提及的条件。然而,它仅限于针对单一文档的问题,忽略了可能需要跨文档推理和优化的更困难情形,例如“最多可以获得多少项奖学金?”这类涉及多个文档的问题不仅因为需要理解更多上下文而更具挑战性,还因为模型必须:(1)探索未提及条件的所有可能组合;(2)理解跨文档条件之间的关系,从而推理出最优结果。为评估模型回答此类问题的能力,我们提出了一个新数据集 MDCR,它能够反映现实世界挑战,并作为需要优化的复杂条件推理的新测试平台。我们使用最新的 LLM 对该数据集进行评估,并展示它们在解决该任务方面的局限性。我们相信该数据集将促进未来关于在条件未知情况下回答优化问题的研究。

英文摘要:

The same real-life questions posed to different individuals may lead to different answers based on their unique situations. For instance, whether a student is eligible for a scholarship depends on eligibility conditions, such as major or degree required. ConditionalQA was proposed to evaluate models' capability of reading a document and answering eligibility questions, considering unmentioned conditions. However, it is limited to questions on single documents, neglecting harder cases that may require cross-document reasoning and optimization, for example, "What is the maximum number of scholarships attainable?" Such questions over multiple documents are not only more challenging due to more context having to understand, but also because the model has to (1) explore all possible combinations of unmentioned conditions and (2) understand the relationship between conditions across documents, to reason about the optimal outcome. To evaluate models' capability of answering such questions, we propose a new dataset MDCR, which can reflect real-world challenges and serve as a new test bed for complex conditional reasoning that requires optimization. We evaluate this dataset using the most recent LLMs and demonstrate their limitations in solving this task. We believe this dataset will facilitate future research in answering optimization questions with unknown conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑