arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.04080cs.DB
收藏

MojoFrame: Mojo语言中的数据框库

MojoFrame: Dataframe Library in Mojo Language

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Shengya Huang, Zhaoheng Li, Derek Warner, Yongjoo Park

AI总结:

本文提出了首个Mojo原生数据框库MojoFrame,利用张量和基数感知方法实现关系操作和用户定义函数,在TPC-H和TPC-DS查询中相比其他语言的数据框库最高加速4.60倍。

摘要:

Mojo是一种新兴的编程语言,基于MLIR(多层中间表示)构建并支持JIT(即时)编译。它能够实现透明的硬件特定优化(例如针对CPU和GPU),同时允许用户使用类似Python的友好语法表达逻辑。Mojo在张量操作上展现了强大的性能;然而,其在数据科学工作流中常见的关系操作(例如过滤、连接和分组聚合)的能力尚未被探索。迄今为止,Mojo生态系统中没有数据框实现。 在本文中,我们介绍了首个Mojo原生数据框库,称为MojoFrame,它支持核心关系操作和用户定义函数(UDF)。MojoFrame构建在Mojo的张量之上,以实现对数值列的快速操作,同时利用基数感知方法有效集成非数值列以实现灵活的数据表示。为了实现高效率,MojoFrame采用了与现有库显著不同的方法。我们展示了MojoFrame支持TPC-H查询的所有操作以及选定的TPC-DS查询,并具有有前途的性能,相比其他编程语言中的现有数据框库实现了高达4.60倍的加速。尽管如此,MojoFrame(以及Mojo语言)仍存在优化机会,特别是在内存数据表示和字典操作方面。

英文摘要:

Mojo is an emerging programming language built on MLIR (Multi-Level Intermediate Representation) and supports JIT (Just-in-Time) compilation. It enables transparent hardware-specific optimizations (e.g., for CPUs and GPUs), while allowing users to express their logic using Python-like user-friendly syntax. Mojo has demonstrated strong performance on tensor operations; however, its capabilities for relational operations (e.g., filtering, join, and group-by aggregation) common in data science workflows, remain unexplored. To date, no dataframe implementation exists in the Mojo ecosystem. In this paper, we introduce the first Mojo-native dataframe library, called MojoFrame, that supports core relational operations and user-defined functions (UDFs). MojoFrame is built on top of Mojo's tensor to achieve fast operations on numeric columns, while utilizing a cardinality-aware approach to effectively integrate non-numeric columns for flexible data representation. To achieve high efficiency, MojoFrame takes significantly different approaches than existing libraries. We show that MojoFrame supports all operations for TPC-H queries and a selection of TPC-DS queries with promising performance, achieving up to 4.60x speedup versus existing dataframe libraries in other programming languages. Nevertheless, there remain optimization opportunities for MojoFrame (and the Mojo language), particularly in in-memory data representation and dictionary operations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑