工作场所中的大型语言模型：面向职位类型分类的提示工程案例研究
Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification
AI总结：
本文以真实职位分类任务为案例，研究提示工程对GPT-3.5系列模型的影响，发现精心设计的零样本gpt-3.5-turbo可超越监督式模型，且提示措辞对性能至关重要。
摘要：
本案例研究调查了真实场景中的职位分类任务，其目标是判断一则英语职位发布是否适合毕业生或入门级岗位。我们探索了多种文本分类方法，包括监督式方法，例如支持向量机（SVM）等传统模型以及DeBERTa等最先进的深度学习方法。我们将它们与在少样本和零样本分类设置下使用的大型语言模型（LLMs）进行比较。为完成该任务，我们采用提示工程，这是一种通过设计提示来引导LLMs产生期望输出的技术。具体而言，我们评估了两个商用且最先进的基于GPT-3.5的语言模型text-davinci-003和gpt-3.5-turbo的性能。我们还详细分析了提示工程的不同方面对模型性能的影响。结果表明，在精心设计的提示下，零样本gpt-3.5-turbo分类器优于所有其他模型，与最佳监督式方法相比，在95%召回率下的精确率提高了6%。此外，我们观察到提示的措辞是激发模型进行适当“推理”的关键因素，而且提示中看似微小的方面也会显著影响模型性能。
英文摘要：
This case study investigates the task of job classification in a real-world setting, where the goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position. We explore multiple approaches to text classification, including supervised approaches such as traditional models like Support Vector Machines (SVMs) and state-of-the-art deep learning methods such as DeBERTa. We compare them with Large Language Models (LLMs) used in both few-shot and zero-shot classification settings. To accomplish this task, we employ prompt engineering, a technique that involves designing prompts to guide the LLMs towards the desired output. Specifically, we evaluate the performance of two commercially available state-of-the-art GPT-3.5-based language models, text-davinci-003 and gpt-3.5-turbo. We also conduct a detailed analysis of the impact of different aspects of prompt engineering on the model's performance. Our results show that, with a well-designed prompt, a zero-shot gpt-3.5-turbo classifier outperforms all other models, achieving a 6% increase in Precision@95% Recall compared to the best supervised approach. Furthermore, we observe that the wording of the prompt is a critical factor in eliciting the appropriate "reasoning" in the model, and that seemingly minor aspects of the prompt significantly affect the model's performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
