arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04466cs.CLcs.AI
收藏

微领域适应预训练在现实操作中是否有效?多步骤评估揭示了潜力和瓶颈

Is Micro Domain-Adaptive Pre-Training Effective for Real-World Operations? Multi-Step Evaluation Reveals Potential and Bottlenecks

Research & Development Group, Hitachi, Ltd(日立株式会社研发部)

Masaya Tsunokake, Yuta Koreeda, Terufumi Morishita, Koichi Nagatsuka, Hikaru Tomonari, Yasuhiro Sogawa

AI总结:

本文通过多步骤评估揭示了微领域适应预训练在生成任务中的潜力和瓶颈,发现其在信息提取方面有效但推理能力有待提升。

摘要:

当将大语言模型 (LLM) 应用于现实中的企业操作时,LLM 需要处理特定操作的小领域(称为微领域)中的专有知识。一项先前研究表明,使用较少文档的微领域适应预训练(mDAPT)效果与在更大领域中使用 DAPT 类似。然而,该研究仅在多项选择题上评估了 mDAPT,因此其在现实操作中生成任务的有效性仍不清楚。我们旨在揭示 mDAPT 在生成任务中的潜力和瓶颈。为此,我们将回答过程分解为三个子任务并评估每个子任务的表现:(1)从 LLM 自己的知识中提取与问题相关的信息;(2)对信息进行推理以获得结论;(3)基于结论组成长文本回答。我们验证了 mDAPT 在现实问题中的 IT 产品知识上。结果表明,mDAPT 解决了基础模型难以解决的提取任务,但未解决其他子任务。这明确了 mDAPT 在知识方面的有效性及其在其他方面的瓶颈。进一步分析表明,解决提取和推理任务确保了足够的性能(超过 90%),强调了增强推理能力的必要性。

英文摘要:

When applying LLMs to real-world enterprise operations, LLMs need to handle proprietary knowledge in small domains of specific operations ($\textbf{micro domains}$). A previous study shows micro domain-adaptive pre-training ($\textbf{mDAPT}$) with fewer documents is effective, similarly to DAPT in larger domains. However, it evaluates mDAPT only on multiple-choice questions; thus, its effectiveness for generative tasks in real-world operations remains unknown. We aim to reveal the potential and bottlenecks of mDAPT for generative tasks. To this end, we disentangle the answering process into three subtasks and evaluate the performance of each subtask: (1) $\textbf{eliciting}$ facts relevant to questions from an LLM's own knowledge, (2) $\textbf{reasoning}$ over the facts to obtain conclusions, and (3) $\textbf{composing}$ long-form answers based on the conclusions. We verified mDAPT on proprietary IT product knowledge for real-world questions in IT technical support operations. As a result, mDAPT resolved the elicitation task that the base model struggled with but did not resolve other subtasks. This clarifies mDAPT's effectiveness in the knowledge aspect and its bottlenecks in other aspects. Further analysis empirically shows that resolving the elicitation and reasoning tasks ensures sufficient performance (over 90%), emphasizing the need to enhance reasoning capability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑