Threadle:一种内存高效的网络存储和查询引擎,适用于大规模、多层和混合模式网络
Threadle: A Memory-Efficient Network Storage and Query Engine for Large, Multilayer, and Mixed-mode Networks
AI总结:
Threadle是一种内存高效的网络存储和查询引擎,通过伪投影方法处理大规模多层混合模式网络,实现高达2000:1的压缩比。
摘要:
我们介绍了Threadle,一种用C#编写的开源、高性能且内存高效的网络存储和查询引擎。该引擎专为处理源自行政登记数据的完整人口网络而设计,这些网络代表了具有数百万节点和数十亿边的非常大规模、多层、混合模式网络。Threadle解决了现有网络库的一个根本限制:无法在大规模上高效处理双模式(二分图)数据。Threadle的核心创新是一种伪投影方法,允许双模式层被查询如同被投影为单模式形式,而无需实际生成内存消耗极大的投影。我们证明,一个包含2000万个节点的网络,其层相当于8万亿条投影边,可以存储在约20GB的RAM中——与实际投影相比,压缩比超过2000:1。此外,Threadle还提供对多层混合模式网络的原生支持,一个集成的节点属性管理器,以及一个带有50多个命令的CLI前端,用于构建、处理、文件处理和管理非常大的异构网络。Threadle在https://www.threadle.dev上免费提供,可以作为预编译二进制文件获取,适用于Windows、macOS和Linux,或直接从源代码编译。Threadle的补充是threadleR,一个R前端,使用户能够对非常大的、异构的、多层、混合模式的群体规模网络进行高级采样和遍历分析。
英文摘要:
We present Threadle, an open-source, high-performance, and memory-efficient network storage and query engine written in C#. Designed for working with full-population networks derived from administrative register data, which represent very large, multilayer, mixed-mode networks with millions of nodes and billions of edges, Threadle addresses a fundamental limitation of existing network libraries: the inability to efficiently handle two-mode (bipartite) data at scale. Threadle's core innovation is a pseudo-projection approach that allows two-mode layers to be queried as if they were projected into one-mode form, without ever materializing the memory-prohibitive projection. We demonstrate that a network with 20 million nodes containing layers equivalent to 8 trillion projected edges can be stored in approximately 20 GB of RAM -- a compression ratio exceeding 2000:1 compared to materialized projection. Additionally, Threadle provides native support for multilayer mixed-mode networks, an integrated node attribute manager, and a CLI frontend with 50+ commands for the construction, processing, file handling, and management of very large heterogeneous networks. Threadle is freely available at https://www.threadle.dev and can either be obtained as precompiled binaries for Win, macOS and Linux, or compiled directly from source. Supplementing Threadle is threadleR, an R frontend that enables advanced sampling- and traversal-based analyses on very large, heterogeneous, multilayer, mixed-mode population-scale networks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。