AutoPureData：自动过滤不良网络数据以更新LLM知识
AutoPureData: Automated Filtering of Undesirable Web Data to Update LLM Knowledge
AI总结：
本文提出AutoPureData系统，自动收集并净化网络数据，以97%和86%的准确率分别消除不安全和不理想文本，确保仅用纯净数据更新LLM知识，支持跨语言RAG检索，提升模型可靠性和安全性。
摘要：
最新且可靠的语言模型一直备受追求，并在各种应用中至关重要。通常，模型在固定数据集上训练后部署到全球。然而，模型的知识会过时。利用网络数据自动更新AI知识涉及重大安全性和质量问题，因为网络中存在不安全和不理想的文本威胁。新数据的纯净性对于更新语言模型知识以保持其可靠性至关重要。本文提出AutoPureData系统，自动收集和净化网络数据。系统加载网络数据样本，利用现有可信AI模型，成功以97%的准确率消除不安全文本，以86%的准确率消除不理想文本，展示了系统在数据净化方面的有效性。系统确保只有有意义且安全的文本可用于更新LLM知识。纯净文本随后被优化并存储在向量数据库中供未来查询。研究发现LLM可以从向量数据库获取新数据。LLM以英文编写RAG查询，即使用户查询是其他语言，证明系统可以进行跨语言检索。本文提出一种方法，通过确保仅使用净化数据更新LLM知识，来维持最新语言模型的准确性和相关性。这项工作有助于使用有意义且安全的文本更新聊天机器人知识，增强其在各行业的实用性，并可能降低因不安全或不纯净数据导致的输出风险。代码可在github.com/Pro-GenAI/AutoPureData获取。
英文摘要：
Up-to-date and reliable language models are consistently sought after and are essential in various applications. Typically, models are trained on a fixed dataset and then deployed globally. However, the knowledge of the models becomes outdated. Enabling automatic updation of AI knowledge using web data involves significant concerns regarding the model's safety and quality due to a threat from unsafe and undesirable text across the web. The purity of new data was essential for updating knowledge of language models to maintain their reliability. This paper proposes AutoPureData, a system that automatically collects and purifies web data. The system loaded a sample of web data. Utilizing existing trusted AI models, it successfully eliminated unsafe text with an accuracy of 97% and undesirable text with an accuracy of 86%, demonstrating the system's effectiveness in purifying the data. The system ensures that only meaningful and safe text can be used to update LLM knowledge. The pure text was then optimized and stored in a vector database for future querying. It was found that LLM can fetch new data from the vector DB. The LLM writes the RAG query in English, even if the user's query is in another language, proving that the system can perform cross-lingual retrieval. This paper proposes a method to maintain the accuracy and relevance of up-to-date language models by ensuring that only purified data was used to update LLM knowledge. This work contributes to updating knowledge of chatbots using meaningful and safe text, enhancing their utility across various industries, and potentially reducing the risks associated with outputs caused by unsafe or impure data. Code is available at github.com/Pro-GenAI/AutoPureData.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
