Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks
University College London(伦敦大学学院);Google(谷歌);Coactive AI(Coactive人工智能公司);Hugging Face(Hugging Face公司);Facebook AI Research(脸书人工智能研究院);The University of Hong Kong(香港大学)
Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, Douwe Kiela
We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers. Dynatask is integrated with Dynabench, a research platform for rethinking benchmarking in AI that facilitates human and model in the loop data collection and evaluation. To create a task, users only need to write a short task configuration file from which the relevant web interfaces and model hosting infrastructure are automatically generated. The system is available at https://dynabench.org/ and the full library can be found at https://github.com/facebookresearch/dynabench.
SiNFluD: Creating and Evaluating Figurative Language Dataset for Sindhi
Department of Data Science, Quaid-e-Awam University of Engineering, Science;Technology, Nawabshah, Pakistan. SoloGenAI Pvt. Ltd. Department of Information Technology, Faculty of Computing;Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia. SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum \& Minerals, Dhahran, Kingdom of Saudi Arabia. Corresponding Author
RoMath: A Mathematical Reasoning Benchmark in Romanian
POLITEHNICA Bucharest National University of Science and Technology(巴格达理工大学科学与技术国立大学);Interdisciplinary School of Doctoral Studies, University of Bucharest(布加勒斯特大学跨学科博士研究学院)
Mapping global dynamics of benchmark creation and saturation in artificial intelligence
Medical University of Vienna(维也纳医科大学);ITTM S.A.—Information Technology for Translational Medicine(ITTM S.A.——转化医学信息技术公司);University of Oxford(牛津大学);Future of Humanity Institute, University of Oxford(牛津大学人类未来研究所)
Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation
Instituto de Telecomunicações(电信研究所);Unbabel(Unbabel公司);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);Carnegie Mellon University(卡内基梅隆大学);CentraleSupélec, Université Paris-Saclay(巴黎萨克雷大学中央高等电力学院);ELLIS Unit Lisbon(里斯本ELLIS单元)
Proper Noun Diacritization for Arabic Wikipedia: A Benchmark Dataset
Computational Approaches to Modeling Language (CAMeL) Lab(计算语言建模方法(CAMeL)实验室);New York University Abu Dhabi(纽约大学阿布扎克分校);Ain Shams University(艾因夏姆斯大学);Stony Brook University(石溪大学);Department of Artificial Intelligence, University of Malta(人工智能系,马耳他大学)
Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers
Institute for Biomedical Informatics, University of Pennsylvania(宾夕法尼亚大学生物医学信息学研究所);AGH University of Science and Technology(AGH科技大学)
了解机器学习(ML)算法的优点和缺点对于确定其应用范围至关重要。在这里,我们介绍了DIverse和GENerative ML Benchmark(DIGEN)-一个综合数据集的集合,用于对二进制结果分类的机器学习算法进行全面,可重复和可解释的基准测试。DIGEN资源由40个数学函数组成,这些函数将连续特征映射到离散端点,以创建合成数据集。这40个函数是使用启发式算法发现的,该算法旨在最大限度地提高多个流行的机器学习算法之间的性能差异,从而为评估和比较新方法提供了有用的测试套件。访问生成函数有助于理解为什么一种方法与其他算法相比表现不佳,从而提供改进的想法。包含大量文档和分析的资源是开源的,可在GitHub上获得。
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
University of Copenhagen(哥本哈根大学);Mila – Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所);University of Cambridge(剑桥大学);TU Darmstadt(达姆施塔特工业大学);New York University(纽约大学);McGill University(麦吉尔大学)
Creating and Evaluating Code-Mixed Nepali-English and Telugu-English Datasets for Abusive Language Detection Using Traditional and Deep Learning Models
Consolidating and Developing Benchmarking Datasets for the Nepali Natural Language Understanding Tasks
Information and Language Processing Research Lab (ILPRL)(信息与语言处理研究实验室);Kathmandu University(加德满都大学);Institute for Research and Innovation in Intelligent Systems (IRIIS)(智能系统研究与创新研究院)
Document Understanding Dataset and Evaluation (DUDE)
KU Leuven(鲁汶大学);Snowflake(雪花公司);Warsaw University of Technology(华沙理工大学);Computer Vision Center, Universitat Autònoma de Barcelona(巴塞罗那自治大学计算机视觉中心);Jagiellonian University(雅盖隆大学);Adam Mickiewicz University(亚当·密茨凯维奇大学);Instabase;University of La Rochelle(拉罗谢尔大学)
University of Bucharest(布加勒斯特大学);MBZ University of Artificial Intelligence(MBZ人工智能大学);Linköping University(利德大学);University of Central Florida(佛罗里达中央大学)
BioSentVec: creating sentence embeddings for biomedical texts
National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH)(美国国立卫生研究院国家医学图书馆国家生物技术信息中心)