Giovanni de Felice, Elena Di Lavore, Mario Román, Alexis Toumi
We present some categorical investigations into Wittgenstein's language-games, with applications to game-theoretic pragmatics and question-answering in natural language processing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
Distributional compositional (DisCo) models are functors that compute the meaning of a sentence from the meaning of its words. We show that DisCo models in the category of sets and relations correspond precisely to relational databases. As a consequence, we get complexity-theoretic reductions from semantics and entailment of a fragment of natural language to evaluation and containment of conjunctive queries, respectively. Finally, we define question answering as an NP-complete problem.
We propose the new problem of learning to recover reasoning chains from weakly supervised signals, i.e., the question-answer pairs. We propose a cooperative game approach to deal with this problem, in which how the evidence passages are selected and how the selected passages are connected are handled by two models that cooperate to select the most confident chains from a large set of candidates (from distant supervision). For evaluation, we created benchmarks based on two multi-hop QA datasets, HotpotQA and MedHop; and hand-labeled reasoning chains for the latter. The experimental results demo…
In this paper, we introduce LiveQA, a new question answering dataset constructed from play-by-play live broadcast. It contains 117k multiple-choice questions written by human commentators for over 1,670 NBA games, which are collected from the Chinese Hupu (https://nba.hupu.com/games) website. Derived from the characteristics of sports games, LiveQA can potentially test the reasoning ability across timeline-based live broadcasts, which is challenging compared to the existing datasets. In LiveQA, the questions require understanding the timeline, tracking events or doing mathematical computations…
In categorical compositional semantics of natural language one studies functors from a category of grammatical derivations (such as a Lambek pregroup) to a semantic category (such as real vector spaces). We compositionally build game-theoretic semantics of sentences by taking the semantic category to be the category whose morphisms are open games. This requires some modifications to the grammar category to compensate for the failure of open games to form a compact closed category. We illustrate the theory using simple examples of Wittgenstein's language-games.
Twenty questions is a widely popular verbal game. In recent years, many computerized versions of this game have been developed in which a user thinks of an entity and a computer attempts to guess this entity by asking a series of boolean-type (yes/no) questions. In this research, we aim to reverse this game by making the computer choose an entity at random. The human aims to guess this entity by quizzing the computer with natural language queries which the computer will then attempt to parse using a boolean question answering model. The game ends when the human is successfully able to guess th…
We designed and built a game called \textit{Immersive Text Game}, which allows the player to choose a story and a character, and interact with other characters in the story in an immersive manner of dialogues. The game is based on several latest models, including text generation language model, information extraction model, commonsense reasoning model, and psychology evaluation model. In the past, similar text games usually let players choose from limited actions instead of answering on their own, and not every time what characters said are determined by the player. Through the combination of …
A Russian Jeopardy! Data Set for Question-Answering Systems
本文介绍了从俄罗斯官方智力竞赛数据库Chgk构建的类Jeopardy!俄语问答数据集，含37万余道题，分析其语言特征并展望相关问答竞赛前景。
We exploit the redundancy and volume of information on the web to build a computerized player for the ABC TV game show 'Who Wants To Be A Millionaire?' The player consists of a question-answering module and a decision-making module. The question-answering module utilizes question transformation techniques, natural language parsing, multiple information retrieval algorithms, and multiple search engines; results are combined in the spirit of ensemble learning using an adaptive weighting scheme. Empirically, the system correctly answers about 75% of questions from the Millionaire CD-ROM, 3rd edit…
Given questions regarding some prototypical situation such as Name something that people usually do before they leave the house for work? a human can easily answer them via acquired experiences. There can be multiple right answers for such questions, with some more common for a situation than others. This paper introduces a new question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. The training set is gathered from an existing set of questions played in a long-running international game show…
Text-based adventure games provide a platform on which to explore reinforcement learning in the context of a combinatorial action space, such as natural language. We present a deep reinforcement learning architecture that represents the game state as a knowledge graph which is learned during exploration. This graph is used to prune the action space, enabling more efficient exploration. The question of which action to take can be reduced to a question-answering task, a form of transfer learning that pre-trains certain parts of our architecture. In experiments using the TextWorld framework, we s…
Scholastic trivia competitions test knowledge and intelligence through mastery of question answering. Modern question answering benchmarks are one variant of the Turing test. Specifically, answering a set of questions as well as a human is a minimum bar towards demonstrating human-like intelligence. This paper makes the case that the format of one competition -- where participants can answer in the middle of hearing a question (incremental) -- better differentiates the skill between (human or machine) players. Additionally, merging a sequential decision-making sub-task with question answering …
Humans observe and interact with the world to acquire knowledge. However, most existing machine reading comprehension (MRC) tasks miss the interactive, information-seeking component of comprehension. Such tasks present models with static documents that contain all necessary information, usually concentrated in a single short substring. Thus, models can achieve strong performance through simple word- and phrase-based pattern matching. We address this problem by formulating a novel text-based question answering task: Question Answering with Interactive Text (QAit). In QAit, an agent must interac…
The Consensus Game: Language Model Generation via Equilibrium Search
本文提出一种无需训练的博弈论解码方法，通过将语言模型生成视为共识博弈并寻找均衡，显著提升多种任务的性能，使小模型超越大模型。
S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering
提出S3MEM框架，通过结构化场景-事件记忆和锚点敏感检索，在长时域交互式问答中实现比通用记忆接口更优的准确率-效率平衡。
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our syst…
We provide a dataset that enables the creation of learning agents that can build knowledge graph-based world models of interactive narratives. Interactive narratives -- or text-adventure games -- are partially observable environments structured as long puzzles or quests in which an agent perceives and interacts with the world purely through textual natural language. Each individual game typically contains hundreds of locations, characters, and objects -- each with their own unique descriptions -- providing an opportunity to study the problem of giving language-based agents the structured memor…
Text adventure games, in which players must make sense of the world through text descriptions and declare actions through text descriptions, provide a stepping stone toward grounding action in language. Prior work has demonstrated that using a knowledge graph as a state representation and question-answering to pre-train a deep Q-network facilitates faster control policy transfer. In this paper, we explore the use of knowledge graphs as a representation for domain knowledge transfer for training text-adventure playing reinforcement learning agents. Our methods are tested across multiple compute…
Knowledge-enhanced Agents for Interactive Text Games
针对文本游戏中智能体连贯性、上下文感知和环境学习不足的问题，本文提出知识注入框架，将正确动作记忆与对象可供性融入强化学习及语言模型智能体，并在 ScienceWorld 上验证其效果。
DragonVerseQA: Open-Domain Long-Form Context-Aware Question-Answering
本文提出开放域长篇问答数据集DragonVerseQA，整合多来源多维度的《权力的游戏》等剧集上下文，通过严格预处理提供长篇答案，为会话AI和叙事分析等任务设定了新的质量基准。
AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
通过问答游戏实验，研究人类在何时以及为何选择委托AI或采纳其建议，发现人类存在对AI正确建议的低依赖（3.9%）和错误建议的过度依赖（1.7%），并受确认偏见影响，建议通过校准置信度、基于证据的解释和信任细化机制来改进人机协作。
Human conversation is a complex mechanism with subtle nuances. It is hence an ambitious goal to develop artificial intelligence agents that can participate fluently in a conversation. While we are still far from achieving this goal, recent progress in visual question answering, image captioning, and visual question generation shows that dialog systems may be realizable in the not too distant future. To this end, a novel dataset was introduced recently and encouraging results were demonstrated, particularly for question answering. In this paper, we demonstrate a simple symmetric discriminative …
Reinforcement learning (RL) has recently shown impressive performance in complex game AI and robotics tasks. To a large extent, this is thanks to the availability of simulated environments such as OpenAI Gym, Atari Learning Environment, or Malmo which allow agents to learn complex tasks through interaction with virtual environments. While RL is also increasingly applied to natural language processing (NLP), there are no simulated textual environments available for researchers to apply and consistently benchmark RL on NLP tasks. With the work reported here, we therefore release NLPGym, an open-…
Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models
本文提出以交际成功作为奖励信号训练交互式语言模型，在纯语言问答设定中验证该奖励可间接反映语法性，并通过强化学习实验发现认知合理约束能改变说话者行为，但尚未提升语言评估表现。
AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy, but the rich variety of standardized exams has remained a landmark challenge. Even in 2016, the best AI system achieved merely 59.3% on an 8th Grade science exam challenge. This paper reports unprecedented success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam's non-diagram, multiple choice (NDMC) questions. In addition, our Aristo system, building upon the success of recent language models, exceeded 83% on the corresponding Grade 12 S…
Are Large Vision Language Models Good Game Players?
针对现有LVLMs评估基准的不足，提出基于游戏的评估框架\method{}，通过四个核心任务全面评估模型在结构化环境中的认知推理能力，并实验揭示了模型在长结构化输出和密集元素感知上的局限。
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Overall, our experiments show tha…
This paper presents a web-based demonstration of Emotion Twenty Questions (EMO20Q), a dialog game whose purpose is to study how people describe emotions. EMO20Q can also be used to develop artificially intelligent dialog agents that can play the game. In previous work, an EMO20Q agent used a sequential Bayesian machine learning model and could play the question-asking role. Newer transformer-based neural machine learning models have made it possible to develop an agent for the question-answering role. This demo paper describes the recent developments in the question-answering role of the EMO20…
This thesis develops the translation between category theory and computational linguistics as a foundation for natural language processing. The three chapters deal with syntax, semantics and pragmatics. First, string diagrams provide a unified model of syntactic structures in formal grammars. Second, functors compute semantics by turning diagrams into logical, tensor, neural or quantum computation. Third, the resulting functorial models can be composed to form games where equilibria are the solutions of language processing tasks. This framework is implemented as part of DisCoPy, the Python lib…
Text-based games provide an interactive way to study natural language processing. While deep reinforcement learning has shown effectiveness in developing the game playing agent, the low sample efficiency and the large action space remain to be the two major challenges that hinder the DRL from being applied in the real world. In this paper, we address the challenges by introducing world-perceiving modules, which automatically decompose tasks and prune actions by answering questions about the environment. We then propose a two-phase training framework to decouple language learning from reinforce…
As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine's ability to reason about language and vision. We descr…
Video game testing requires game-specific knowledge as well as common sense reasoning about the events in the game. While AI-driven agents can satisfy the first requirement, it is not yet possible to meet the second requirement automatically. Therefore, video game testing often still relies on manual testing, and human testers are required to play the game thoroughly to detect bugs. As a result, it is challenging to fully automate game testing. In this study, we explore the possibility of leveraging the zero-shot capabilities of large language models for video game bug detection. By formulatin…
Skin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMs
本文提出 Skin-in-the-Game (SKIG) 框架，通过从多利益相关者视角探索决策后果、模拟问责机制、共情练习和风险评估，有效增强了大型语言模型在复杂场景下的道德推理与伦理决策能力。
Complete Chess Games Enable LLM Become A Chess Master
本文提出 ChessLLM 模型，通过监督微调将国际象棋转化为文本格式，在与 Stockfish 对弈中达到 1788 的专业级 Elo 评分，并验证了长回合数据监督的优势。
Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Game
本研究以狼人杀游戏为平台，通过整合警长角色框架并设计可靠性与影响力两项新指标，结合自建WWQA数据集与人类参与者实验，全面评估了不同规模LLMs的意见领导力，发现仅少数模型具备该能力。
Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model
本文系统评估多种检索器增强语言模型的陈述推理能力，揭示检索不全、模型推理弱及二者误差叠加问题，并分析多跳检索并阅读在不同规模模型上的泛化差异。
Prior work on training generative Visual Dialog models with reinforcement learning(Das et al.) has explored a Qbot-Abot image-guessing game and shown that this 'self-talk' approach can lead to improved performance at the downstream dialog-conditioned image-guessing task. However, this improvement saturates and starts degrading after a few rounds of interaction, and does not lead to a better Visual Dialog model. We find that this is due in part to repeated interactions between Qbot and Abot during self-talk, which are not informative with respect to the image. To improve this, we devise a simpl…
Nash Equilibrium Text: A Game-Theoretic Decoding Framework for Text Generation
本文提出一种基于博弈论的文本生成解码框架，将修订建模为纳什均衡，并设计纳什解码算法，在问答基准上无需微调即可显著超越更大规模的自回归模型。
LLMs as Agentic Cooperative Players in Multiplayer UNO
研究让LLM作为合作玩家在UNO中协助他人获胜，发现尽管不同规模模型均超越随机基线，但鲜少能显著帮助队友。
Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
本文提出以20个问题式的实体推断游戏作为评估框架，系统探究大语言模型的多轮对话推理与规划能力，发现GPT-4等强模型大幅超越人类，并通过行为克隆与强化学习显著提升了较弱模型的表现。
While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills. In this work, we introduce WinoGAViL: an online game of vision-and-language associations (e.g., between werewolves and a full moon), used as a dynamic evaluation benchmark. Inspired by the popular card game Codenames, a spymaster gives a textual cue related to several visual candidates, and another player tries to identify them. Human players are rewarded for creating associations that are challenging for a rival AI model but still s…
HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies
本文提出合作博弈 HiddenTables 和数据集 PyQTax，通过自然语言模式让 LLM 在保护数据隐私的前提下解决表格问答任务，并揭示现有 LLM 在复杂查询上的不足。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。