高效数据帧系统:瘦身的惰性Fat Pandas
Efficient Dataframe Systems: Lazy Fat Pandas on a Diet
AI总结:
针对Pandas处理超内存数据集困难、现有惰性框架需重写代码的问题,提出仅需改两行代码的框架,结合即时静态分析与惰性数据帧包装的运行时优化,支持多后端选择,提升了大数据处理性能。
摘要:
Pandas被广泛应用于数据科学领域,但当数据集规模超出内存时,用户常会遇到问题。目前已有多种基于惰性求值的框架可处理大型数据集,但用户必须重写程序以适配这些框架,且多框架并存的情况也增加了程序员的工作复杂度。本文提出了一个框架,允许程序员使用原生Pandas编写代码;用户仅需修改两行代码,我们的系统便会结合即时静态分析与基于惰性数据帧包装框架的运行时优化来对程序进行优化。此外,该系统允许程序员选择后端,可与Pandas、Dask和Modin无缝协作,支持根据数据规模等因素为应用选择最合适的后端。在多种程序上的性能测试结果验证了本框架的优势。
英文摘要:
Pandas is widely used for data science applications, but users often run into problems when datasets are larger than memory. There are several frameworks based on lazy evaluation that handle large datasets, but the programs have to be rewritten to suit the framework, and the presence of multiple frameworks complicates the life of a programmer. In this paper we present a framework that allows programmers to code in plain Pandas; with just two lines of code changed by the user, our system optimizes the program using a combination of just-in-time static analysis, and runtime optimization based on a lazy dataframe wrapper framework. Moreover, our system allows the programmer to choose the backend. It works seamlessly with Pandas, Dask, and Modin, allowing the choice of the best-suited backend for an application based on factors such as data size. Performance results on a variety of programs show the benefits of our framework.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。