Reinforcement learning (RL) has recently shown impressive performance in complex game AI and robotics tasks. To a large extent, this is thanks to the availability of simulated environments such as OpenAI Gym, Atari Learning Environment, or Malmo which allow agents to learn complex tasks through interaction with virtual environments. While RL is also increasingly applied to natural language processing (NLP), there are no simulated textual environments available for researchers to apply and consistently benchmark RL on NLP tasks. With the work reported here, we therefore release NLPGym, an open-source Python toolkit that provides interactive textual environments for standard NLP tasks such as sequence tagging, multi-label classification, and question answering. We also present experimental results for 6 tasks using different RL algorithms which serve as baselines for further research. The toolkit is published at https://github.com/rajcscw/nlp-gym
Citekit: A Modular Toolkit for Large Language Model Citation Generation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
We present ConvLab-2, an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems. As the successor of ConvLab (Lee et al., 2019b), ConvLab-2 inherits ConvLab's framework but integrates more powerful dialogue models and supports more datasets. Besides, we have developed an analysis tool and an interactive tool to assist researchers in diagnosing dialogue systems. The analysis tool presents rich statistics and summarizes common mistakes from simulated dialogues, which…
We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for e…
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Lab of Artificial Intelligence for Education, East China Normal University（华东师范大学教育人工智能实验室）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海教育人工智能研究院）
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Massachusetts Institute of Technology（麻省理工学院）；University of Waterloo（滑铁卢大学）；Meta AI；AIRI（人工智能研究院）；École Polytechnique de Montréal（蒙特利尔理工学院）；Microsoft Research（微软研究院）
Microsoft（微软）；University of Wisconsin--Madison（威斯康星大学麦迪逊分校）；UCLA（加州大学洛杉矶分校）
Learning visual representations from natural language supervision has recently shown great promise in a number of pioneering works. In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks. However, it remains challenging to evaluate the transferablity of these models due to the lack of easy-to-use evaluation toolkits and public benchmarks. To tackle this, we build ELEVATER (Evaluation of Language-augmented Visual Task-level Transfer), the first benchmark and toolkit for evaluating(pre-trained) language-augmented visual models. EL…
SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs
University of California San Diego（加州大学圣地亚哥分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）；University of California, Los Angeles（加州大学洛杉矶分校）；California Institute of Technology（加州理工学院）；ETH Zurich（苏黎世联邦理工学院）
Evaluating LLM-based Workflows for Switched-Mode Power Supply Design
Robert Bosch GmbH, Cross-Domain Computing Solutions（罗伯特·博世有限公司，跨领域计算解决方案）；University of Stuttgart, Institute for Micro Integration (IFM)（斯图加特大学，微系统集成研究所）；Hahn-Schickard, Stuttgart, Germany（哈恩-施克尔德研究所，斯图加特，德国）
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Stanford University（斯坦福大学）
Textual adversarial attacking has received wide and increasing attention in recent years. Various attack models have been proposed, which are enormously distinct and implemented with different programming frameworks and settings. These facts hinder quick utilization and fair comparison of attack models. In this paper, we present an open-source textual adversarial attack toolkit named OpenAttack to solve these issues. Compared with existing other textual adversarial attack toolkits, OpenAttack has its unique strengths in support for all attack types, multilinguality, and parallel processing. Cu…
Open-Source Conversational AI with SpeechBrain 1.0
Concordia University（康考迪亚大学）；Mila-Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Samsung AI Center Cambridge（三星剑桥人工智能中心）；University of Cambridge（剑桥大学）；Avignon University（阿维尼翁大学）；Laval University（拉瓦尔大学）；Zaion（扎伊翁）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）；Aalto University（阿尔托大学）；University of Bologna（博洛尼亚大学）；Telecom Paris（巴黎电信学院）；University of Edinburgh（爱丁堡大学）；Inria（法国国家信息与自动化研究所）；Silo AI（西罗人工智能公司）；Idiap（伊迪亚普研究所）；KU Leuven（鲁汶大学）；EPFL（洛桑联邦理工学院）；University of Trento（特伦托大学）；National Institute of Informatics - Tokyo（东京情报学研究所）
In this paper, we present kogito, an open-source tool for generating commonsense inferences about situations described in text. kogito provides an intuitive and extensible interface to interact with natural language generation models that can be used for hypothesizing commonsense knowledge inference from a textual input. In particular, kogito offers several features for targeted, multi-granularity knowledge generation. These include a standardized API for training and evaluating knowledge models, and generating and filtering inferences from them. We also include helper functions for converting…
Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Models
AIM Intelligence；Seoul National University（首尔国立大学）；A.I.MATICS；TigerCompany；Catius；National Assembly of Korea（韩国国会）；Coupang；Hankuk University of Foreign Studies（韩国外交大学）
CogBench: a large language model walks into a psychology lab
Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）；Helmholtz Computational Health Center（亥姆霍兹计算健康中心）；Institute for Human-Centered AI（以人为本人工智能研究所）；Google DeepMind（谷歌DeepMind）
Rationality Check! Benchmarking the Rationality of Large Language Models
Department of Electronic Engineering, BNRist, Tsinghua University（电子工程系、BNRist、清华大学）；Department of Sociology, University of Chicago（社会学系、芝加哥大学）；Santa Fe Institute（圣菲研究所）
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Westlake University（西湖大学）；The University of Hong Kong（香港大学）；HKUST（香港科技大学）；Zhejiang University（浙江大学）
Despite impressive performance on standard benchmarks, deep neural networks are often brittle when deployed in real-world systems. Consequently, recent research has focused on testing the robustness of such models, resulting in a diverse set of evaluation methodologies ranging from adversarial attacks to rule-based data transformations. In this work, we identify challenges with evaluating NLP systems and propose a solution in the form of Robustness Gym (RG), a simple and extensible evaluation toolkit that unifies 4 standard evaluation paradigms: subpopulations, transformations, evaluation sets…
University of Zurich（苏黎世大学）；University of Dayton（Dayton 大学）；University of Surrey（萨里大学）；Idiap Research Institute（Idiap 研究院）；EPFL（苏黎世联邦理工学院）；Google DeepMind（谷歌DeepMind）
DFKI GmbH（德国人工智能研究中心）；Università di Bologna（博洛尼亚大学）；Qatar Computing Research Institute (QCRI-HBKU)（卡塔尔计算研究所（QCRI-HBKU））；Amazon（亚马逊）
We propose an automatic language-independent graph-based method to build à-la-carte article collections on user-defined domains from the Wikipedia. The core model is based on the exploration of the encyclopaedia's category graph and can produce both monolingual and multilingual comparable collections. We run thorough experiments to assess the quality of the obtained corpora in 10 languages and 743 domains. According to an extensive manual evaluation, our graph-based model outperforms a retrieval-based approach and reaches an average precision of 84% on in-domain articles. As manual evaluations…
Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；BNRist, Tsinghua University（清华大学BNRist）
