Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective. However, development sets are widely used in research papers that purport to deal with low-resource natural language processing (NLP). Here, we aim to answer the following questions: Does using a development set for early stopping in the low-resource setting influence results as compared to a more realistic alternative, where the number of training epochs is tuned on development languages? And does it lead to overestimation or underestimation of performance? We repeat multiple experiments from recent work on neural models for low-resource NLP and compare results for models obtained by training with and without development sets. On average over languages, absolute accuracy differs by up to 1.4%. However, for some languages and tasks, differences are as big as 18.0% accuracy. Our results highlight the importance of realistic experimental setups in the publication of low-resource NLP research results.
Toward a realistic model of speech processing in the brain with self-supervised learning
Ecole Normale Supérieure, PSL University（巴黎高等师范学院，巴黎文理研究大学）；Université de Paris cité（巴黎城市大学）；Université Paris-Saclay（巴黎萨克雷大学）；Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）；University of Toronto（多伦多大学）；INSERM（法国国家健康与医学研究院）；Meta AI
SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence Shanghai Jiao Tong University（X-LANCE实验室，计算机科学学院，人工智能教育部重点实验室，上海交通大学）；Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）；Peng Cheng Laboratory（鹏城实验室）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Tianjin University（天津大学）；Hong Kong University of Science and Technology（香港科学大学）；Queen Mary University of London（伦敦玛丽女王大学）；Nanyang Technological University（南洋理工大学）；Shanghai Innovation Institute（上海创新研究院）
Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark
The Sixth Affiliated Hospital of Sun Yat-sen University（中山大学第六附属医院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济发展实验室（深圳））；Xinyi People’s Hospital（新一人民医院）；The Fifth Affiliated Hospital of Sun Yat-sen University（中山大学第五附属医院）；School of Public Health of Sun Yat-sen University（中山大学公共卫生学院）
Artificial Intelligence, speech and language processing approaches to monitoring Alzheimer's Disease: a systematic review
Usher Institute, Edinburgh Medical School, The University of Edinburgh（爱丁堡大学爱丁堡医学院厄舍研究所）；Centre for Dementia Prevention, The University of Edinburgh（爱丁堡大学痴呆预防中心）
语言是阿尔茨海默病临床信息的宝贵来源，因为它与神经变性同时下降。因此，语音和语言数据已被广泛研究与其诊断。本文总结了当前关于使用人工智能、语音和语言处理来预测阿尔茨海默病背景下认知能力下降的研究结果，详细介绍了当前的研究程序，强调了其局限性，并提出了解决这些问题的策略。我们对2000年至2019年期间在PROSPERO（参考CRD 42018116606）中注册的原始研究进行了系统性综述。跨学科检索涵盖了工程学（ACM和IEEE）、心理学（PsycINFO）、医学（PubMed和Embase）和Web of Science的六个数据库。相关论文的参考书目筛选至2019年12月。根据合格性标准，从3，654个检索结果中选择了51篇文章。四个表格总结了他们的研究结果：研究细节（目的、人群、干预措施、比较、方法和结局）、数据细节（规模、类型、模式、注释、平衡、可用性和研究语言）、方法（预处理、特征生成、机器学习、评价和结果）和临床适用性（研究意义、临床潜力、偏倚风险和优势/局限性）。虽然几乎所有51项研究都报告了有希望的结果，但很少有在临床研究或实践中实施。我们的结论是，该领域的主要局限性是标准化差，结果的可比性有限，以及研究目的和临床应用之间的脱节程度。缩小这些差距的努力应该支持未来研究转化为临床实践。
