超越指标：评估 LLM 在具有文化细微差异的低资源现实场景中的有效性
Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios
AI总结：
本文基于多语言、语码混杂的 WhatsApp 情感分析数据集评估七个主流 LLM，结合 F1 与解释定性分析，指出高指标不等于文化语境理解能力，并呼吁构建相应评估基准。
摘要：
大型语言模型（LLM）在现实应用中的部署既带来机遇也带来挑战，尤其是在多语言和语码混杂的交流环境中。本研究在一个源自多语言、语码混杂 WhatsApp 聊天记录的数据集上评估了七个主流 LLM 的情感分析表现，数据涉及斯瓦希里语、英语和 Sheng。我们的评估既包括使用 F1 分数等指标进行的定量分析，也包括对 LLM 预测解释的定性评价。研究发现，尽管 Mistral-7b 和 Mixtral-8x7b 取得了较高的 F1 分数，但它们以及 GPT-3.5-Turbo、Llama-2-70b 和 Gemma-7b 等其他 LLM 在理解语言和语境细微差异方面存在困难，并且从其解释中可以看出，它们的决策过程缺乏透明度。相比之下，GPT-4 和 GPT-4-Turbo 擅长把握多样化语言输入并处理各类语境信息，在决策过程中表现出与人类对齐的高度一致性及透明度。然而，这些 LLM 在融入文化细微差异方面仍遇到困难，尤其是在非英语环境中，GPT-4 系列模型的表现也不稳定。研究结果强调，有必要持续改进 LLM，以有效应对具有文化细微差异的低资源现实环境所带来的挑战，同时也需要开发能够捕捉这些问题的评估基准。
英文摘要：
The deployment of Large Language Models (LLMs) in real-world applications presents both opportunities and challenges, particularly in multilingual and code-mixed communication settings. This research evaluates the performance of seven leading LLMs in sentiment analysis on a dataset derived from multilingual and code-mixed WhatsApp chats, including Swahili, English and Sheng. Our evaluation includes both quantitative analysis using metrics like F1 score and qualitative assessment of LLMs' explanations for their predictions. We find that, while Mistral-7b and Mixtral-8x7b achieved high F1 scores, they and other LLMs such as GPT-3.5-Turbo, Llama-2-70b, and Gemma-7b struggled with understanding linguistic and contextual nuances, as well as lack of transparency in their decision-making process as observed from their explanations. In contrast, GPT-4 and GPT-4-Turbo excelled in grasping diverse linguistic inputs and managing various contextual information, demonstrating high consistency with human alignment and transparency in their decision-making process. The LLMs however, encountered difficulties in incorporating cultural nuance especially in non-English settings with GPT-4s doing so inconsistently. The findings emphasize the necessity of continuous improvement of LLMs to effectively tackle the challenges of culturally nuanced, low-resource real-world settings and the need for developing evaluation benchmarks for capturing these issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
