数据库中大规模数据分配的建模与优化
Modeling and Optimization for Massive Data Allocation in Database
AI总结:
针对分布式数据库中数据分散导致通信开销大的问题,提出基于谱聚类归一化割的0-1优化模型,并用Bregman近端梯度法求解,实现数据平衡与低通信开销。
摘要:
在大数据时代,电子商务和互联网平台面临处理海量数据的挑战。然而,由于数据分布在分布式数据库的不同机器上,收集相关数据以完成交易会产生额外的通信成本。如果没有精心设计的数据放置方案,这种成本会严重影响在线事务处理系统的性能。为了满足行业需求,迫切需要能够在固定时间内输出实现i)数据平衡和ii)低通信开销的数据放置方案的算法。尽管已有一些方法被研究,但它们未能充分满足上述要求。在本文中,受谱聚类的归一化割启发,我们引入了一种新的数据分配问题模型。归一化割调和了两个目标之间的固有冲突。考虑到模型的变量特性,我们将问题表述为0-1优化问题,并使用保证收敛的Bregman近端梯度法求解松弛问题。数值实验表明,收敛解可以平滑地舍入为离散解。此外,我们的算法在最小化迁移成本的同时保持优越的平衡性,超越了简单的和元启发式的分区方案。
英文摘要:
In the era of big data, e-commerce and Internet platforms face the challenge of processing massive amounts of data. However, due to data being scattered across different machines in distributed database, extra communication costs are incurred in gathering relevant data to complete transactions. Without a carefully designed data placement scheme, this cost can severely impact the performance of Online Transaction Processing systems. To meet industry requirements, algorithms that output a data placement scheme that achieves i) data balance and ii) low communication overhead within a fixed period of time are eagerly investigated. Although some existing methods have been studied, they do not adequately meet the aforementioned requirements. In this paper, inspired by the normalized cut of spectral clustering, we introduce a novel model for data allocation problem. The normalized cut reconciles the inherent conflict between the two objectives. Taking into account the variable characteristics of the model, we formulate the problem as a 0-1 optimization problem, and solve the relaxed problem using the Bregman proximal gradient method with guaranteed convergence. The numerical experiments reveal that the convergent solutions can be smoothly rounded to discrete solutions. Furthermore, our algorithm surpasses both simple and meta-heuristic partitioning schemes by minimizing migration costs while maintaining a superior balance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。