arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.16909cs.DBcs.AI
收藏

RetClean:基于检索的数据清洗方法,利用基础模型和数据湖

RetClean: Retrieval-Based Data Cleaning Using Foundation Models and Data Lakes

QCRI, HBKU(卡塔尔计算研究所,哈马德·本·哈利法大学);HKUST(GZ)/HKUST(香港科技大学(广州)/香港科技大学)

Zan Ahmad Naeem, Mohammad Shahmeer Ahmad, Mohamed Eltabakh, Mourad Ouzzani, Nan Tang

AI总结:

RetClean利用基础模型与数据湖,通过检索增强方式解决数据清洗中的未知数据集与隐私问题,支持ChatGPT辅助、数据湖检索及本地微调三种场景。

摘要:

基础模型(如ChatGPT)能否清洗您的数据?在本提案中,我们证明ChatGPT确实可以通过为数据表中的特定单元格建议修正来协助数据清洗(场景1)。然而,ChatGPT可能难以处理其从未遇到过的数据集(例如,本地企业数据),或者当用户需要解释建议清洗值的来源时。为了解决这些问题,我们开发了一种基于检索的方法,将ChatGPT的能力与用户提供的数据湖相结合。首先对数据湖进行索引,然后检索与用户查询元组最相关的top-k个元组,最后利用ChatGPT推断出正确的值(场景2)。然而,出于隐私原因,将企业数据共享给外部托管的模型ChatGPT可能不可行。为了支持此场景,我们开发了一个基于RoBERTa的自定义基础模型,可以本地部署。通过在小量示例上进行微调,它可以基于检索到的元组有效进行值推断(场景3)。我们提出的系统RetClean无缝支持所有三种场景,并提供用户友好的图形界面,使VLDB受众能够探索和试验该系统。

英文摘要:

Can foundation models (such as ChatGPT) clean your data? In this proposal, we demonstrate that indeed ChatGPT can assist in data cleaning by suggesting corrections for specific cells in a data table (scenario 1). However, ChatGPT may struggle with datasets it has never encountered before (e.g., local enterprise data) or when the user requires an explanation of the source of the suggested clean values. To address these issues, we developed a retrieval-based method that complements ChatGPT's power with a user-provided data lake. The data lake is first indexed, we then retrieve the top-k relevant tuples to the user's query tuple and finally leverage ChatGPT to infer the correct value (scenario 2). Nevertheless, sharing enterprise data with ChatGPT, an externally hosted model, might not be feasible for privacy reasons. To assist with this scenario, we developed a custom RoBERTa-based foundation model that can be locally deployed. By fine-tuning it on a small number of examples, it can effectively make value inferences based on the retrieved tuples (scenario 3). Our proposed system, RetClean, seamlessly supports all three scenarios and provides a user-friendly GUI that enables the VLDB audience to explore and experiment with the system.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑