Code-switching, the alternation of languages within a single discourse, presents a significant challenge for Automatic Speech Recognition. Despite the unique nature of the task, performance is commonly measured with established metrics such as Word-Error-Rate (WER). However, in this paper, we question whether these general metrics accurately assess performance on code-switching. Specifically, using both Connectionist-Temporal-Classification and Encoder-Decoder models, we show fine-tuning on non-code-switched data from both matrix and embedded language improves classical metrics on code-switching test sets, although actual code-switched words worsen (as expected). Therefore, we propose Point-of-Interest Error Rate (PIER), a variant of WER that focuses only on specific words of interest. We instantiate PIER on code-switched utterances and show that this more accurately describes the code-switching performance, showing huge room for improvement in future work. This focused evaluation allows for a more precise assessment of model performance, particularly in challenging aspects such as inter-word and intra-word code-switching.
CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);College of Software, Jilin University(吉林大学软件学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.(中国电信人工智能技术(北京)有限公司星辰AGI实验室);Bilibili Inc.(哔哩哔哩公司);University of Ottawa(渥太华大学);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
Adapting Language Balance in Code-Switching Speech
Interactive Systems Lab, Karlsruhe Institut of Technology (KIT), Germany(互动系统实验室,卡尔斯鲁厄理工学院(KIT),德国);InterACT, Carnegie Mellon University (CMU), USA(InterACT,卡内基梅隆大学(CMU),美国)
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Multimodal Art Projection Research Community(多模态艺术投影研究社区);University of Waterloo(滑铁卢大学);HKUST(香港科技大学);University of Manchester(曼彻斯特大学);Tongji University(同济大学);Vector Institute(矢量研究所)
Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spaces
Carnegie Mellon University(卡内基梅隆大学);Africa Kigali Rwanda(非洲基加利卢旺达);Bahir Dar Institute of Technology(巴希尔达尔技术学院);Federal University Otuoke Bayelsa Nigeria(贝莱萨州联邦大学)