We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data is generated using a massive probabilistic grammar (based on state-split PCFGs), that is itself derived from a large natural language corpus, but also provides us complete control over the generative process. We describe and release both grammar and corpus, and test for the naturalness of our generated data. This approach allows us to define closed-form expressions to efficiently compute exact lower bounds on obtainable perplexity using both causal and masked language modelling. Our results show striking differences between neural language modelling architectures and training objectives in how closely they allow approximating the lower bound on perplexity. Our approach also allows us to directly compare learned representations to symbolic rules in the underlying source. We experiment with various techniques for interpreting model behaviour and learning dynamics. With access to the underlying true source, our results show striking differences and outcomes in learning dynamics between different classes of words.
Automated Speech Scoring System Under The Lens: Evaluating and interpreting the linguistic cues for language proficiency
Columbia University(哥伦比亚大学);Second Language Testing, Inc.(第二语言测试公司);Indraprastha Institute of Information Technology, Delhi(德里印多尔信息科技学院);SUNY at Buffalo(布法罗纽约州立大学);Adobe Media Data Science Research(奥多比媒体数据科学研究中心)
The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping
Multimodal Language Department(多模态语言部门);Max Planck Institute for Psycholinguistics(马克斯·普朗克心理语言学研究所);Department of Linguistics(语言学系);Boğaziçi University(博多伊奇大学);Donders Institute for Brain Cognition and Behaviour(多纳尔斯脑认知与行为研究所);Radboud University(拉德堡德大学);Department of Linguistics and Communication(语言学与沟通系);University of Birmingham(伯明翰大学)
Department of Computer & Information Science & Engineering, University of Florida(佛罗里达大学计算机与信息科学与工程系);School of Computer Science, University of Oklahoma(俄克拉荷马大学计算机科学学院);Department of Electrical Engineering and Computer Science, United States Military Academy(美国军事学院电子工程与计算机科学系)
Maintaining Journalistic Integrity in the Digital Age: A Comprehensive NLP Framework for Evaluating Online News Content
The Institute for Artificial Intelligence Research and Development of Serbia(塞尔维亚人工智能研究与发展研究所);University of Belgrade(贝尔格莱德大学);Universitas Negeri Padang(巴东国立大学)
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);Yale University(耶鲁大学)