The present study asks if ChatGPT4, the version of ChatGPT which uses the language model GPT4, can successfully solve introductory linguistic exams. Previous exam questions of an Introduction to Linguistics course at a German university are used to test this. The exam questions were fed into ChatGPT4 with only minimal preprocessing. The results show that the language model is very successful in the interpretation even of complex and nested tasks. It proved surprisingly successful in the task of broad phonetic transcription, but performed less well in the analysis of morphemes and phrases. In simple cases it performs sufficiently well, but rarer cases, particularly with missing one-to-one correspondence, are currently treated with mixed results. The model is not yet able to deal with visualisations, such as the analysis or generation of syntax trees. More extensive preprocessing, which translates these tasks into text data, allow the model to also solve these tasks successfully.
Jožef Stefan Institute（约泽夫·斯蒂芬研究所）；Center za jezikovne vire in tehnologije（语言资源与技术中心）；Univerze v Ljubljani（卢布尔雅那大学）
ChatGPT has shown strong capabilities in natural language generation tasks, which naturally leads researchers to explore where its abilities end. In this paper, we examine whether ChatGPT can be used for zero-shot text classification, more specifically, automatic genre identification. We compare ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on datasets, manually annotated with genres. The models are compared on test sets in two languages: English and Slovenian. Results show that ChatGPT outperforms the fine-tuned model when applied to the dataset which was not seen…
Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model
Carnegie Mellon University（卡内基梅隆大学）；LMU Munich（慕尼黑大学）；University of Oxford（牛津大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Allen Institute for AI（艾伦人工智能研究所）；IIT Delhi（德里印度理工学院）
A blind spot for large language models: Supradiegetic linguistic information
University of Vermont（佛蒙特大学）；Vermont Advanced Computing Center（佛蒙特高级计算中心）；Vermont Complex Systems Center（佛蒙特复杂系统中心）；MassMutual Center of Excellence in Complex Systems and Data Science（MassMutual复杂系统与数据科学卓越中心）；Department of Computer Science, University of Vermont（佛蒙特大学计算机科学系）
Most online message threads inherently will be cluttered and any new user or an existing user visiting after a hiatus will have a difficult time understanding whats being discussed in the thread. Similarly cluttered responses in a message thread makes analyzing the messages a difficult problem. The need for disentangling the clutter is much higher when the platform where the discussion is taking place does not provide functions to retrieve reply relations of the messages. This introduces an interesting problem to which \cite{wang2011learning} phrases as a structural learning problem. We create…
We asked ChatGPT to participate in an undergraduate computer science exam on ''Algorithms and Data Structures''. The program was evaluated on the entire exam as posed to the students. We hand-copied its answers onto an exam sheet, which was subsequently graded in a blind setup alongside those of 200 participating students. We find that ChatGPT narrowly passed the exam, obtaining 20.5 out of 40 points. This impressive performance indicates that ChatGPT can indeed succeed in challenging tasks like university exams. At the same time, the questions in our exam are structurally similar to those of …
Comparative Analysis of ChatGPT, GPT-4, and Microsoft Bing Chatbots for GRE Test
Thornton Tomasetti, Inc.（Thornton Tomasetti 公司）；Yarmouk University（耶尔穆克大学）；Ministry of Municipal Affairs（市政事务部）；Jordan University of Science and Technology（约旦科技大学）
