This paper presents a systematic review of common analytical data architectures based on DAMA-DMBOK and ArchiMate. The paper is work in progress and provides a first view on Gartner's Logical Data Warehouse paradigm, Data Fabric and Dehghani's Data Mesh proposal as well as their interdependencies. It furthermore sketches the way forward how this work can be extended by covering more architecture paradigms (incl. classic Data Warehouse, Data Vault, Data Lake, Lambda and Kappa architectures) and introducing a template with among others "context", "problem" and "solution" descriptions, leading ultimately to a pattern system providing guidance for choosing the right architecture paradigm for the right situation.
组织经常面临为可能具有不同底层数据和编程模型的大型异构数据集提供数据管理解决方案的挑战。例如，医疗数据集可以具有非结构化文本、关系数据、时间序列波形和图像。试图将这些数据集放入单个数据管理系统中可能会对性能和效率产生不利影响。作为英特尔大数据科学技术中心的一部分，我们正在开发一种针对此类问题设计的聚合存储系统。BigDAWG（Big Data Analytics Working Group的缩写）是一个聚合存储系统，旨在处理自然跨越不同处理或存储引擎的复杂问题。BigDAWG提供了一种体系结构，支持使用不同数据模型的各种数据库系统，通过岛和提供统一多岛接口的中间件支持位置透明性和语义完整性的竞争概念。BigDAWG系统应用于医疗数据集的原型的初步结果验证了polystore概念。在本文中，我们将描述polystore数据库，当前的BigDAWG架构及其在MIMIC II医疗数据集上的应用，初始性能结果以及我们未来的发展计划。
Servers produced by mainstream vendors are inefficient in processing Big Data queries due to bottlenecks inherent in the fundamental architecture of these systems. Current server blades contain multicore processors connected to DRAM memory and disks by an interconnection chipset. The multicore processor chips perform all the computations while the DRAM and disks store the data but have no processing capability. To perform a database query, data must be moved back and forth between DRAM and a small cache as well as between DRAM and disks. For Big Data applications this data movement in onerous.…
One of the purposes of Big Data systems is to support analysis of data gathered from heterogeneous data sources. Since data warehouses have been used for several decades to achieve the same goal, they could be leveraged also to provide analysis of data stored in Big Data systems. The problem of adapting data warehouse data and schemata to changes in these requirements as well as data sources has been studied by many researchers worldwide. However, innovative methods must be developed also to support evolution of data warehouses that are used to analyze data stored in Big Data systems. In this …
Various tools, softwares and systems are proposed and implemented to tackle the challenges in big data on different emphases, e.g., data analysis, data transaction, data query, data storage, data visualization, data privacy. In this paper, we propose datar, a new prospective and unified framework for Big Data Management System (BDMS) from the point of system architecture by leveraging ideas from mainstream computer structure. We introduce five key components of datar by reviewing the cur- rent status of BDMS. Datar features with configuration chain of pluggable engines, automatic dataflow on j…
This paper reviews recent advances in big data optimization, providing the state-of-art of this emerging field. The main focus in this review are optimization techniques being applied in big data analysis environments. Integer linear programming, coordinate descent methods, alternating direction method of multipliers, simulation optimization and metaheuristics like evolutionary and genetic algorithms, particle swarm optimization, differential evolution, fireworks, bat, firefly and cuckoo search algorithms implementations are reviewed and discussed. The relation between big data optimization an…
Over the past two decades, we have witnessed an exponential increase of data production in the world. So-called big data generally come from transactional systems, and even more so from the Internet of Things and social media. They are mainly characterized by volume, velocity, variety and veracity issues. Big data-related issues strongly challenge traditional data management and analysis systems. The concept of data lake was introduced to address them. A data lake is a large, raw data repository that stores and manages all company data bearing any format. However, the data lake concept remains…
With the advent of blockchain technology, multiple research avenues and platforms for dialogue have opened up. However technology transfer to the pubic has not been implemented, such that regular public can access and make use of secure and decentralized software. Most blockchain solutions till date deal with financial applications or monetary transactions, which may not be helpful or be accessible to the general public, especially the lower levels of the financial society. Medi-Chain is a people-first medical blockchain with a usable desktop application and interface which makes use of cuttin…
Apache Hive is an open-source relational database system for analytic big-data workloads. In this paper we describe the key innovations on the journey from batch tool to fully fledged enterprise data warehousing system. We present a hybrid architecture that combines traditional MPP techniques with more recent big data and cloud concepts to achieve the scale and performance required by today's analytic applications. We explore the system by detailing enhancements along four main axis: Transactions, optimizer, runtime, and federation. We then provide experimental results to demonstrate the perfo…
Big Data architectures allow to flexibly store and process heterogeneous data, from multiple sources, in their original format. The structure of those data, commonly supplied by means of REST APIs, is continuously evolving. Thus data analysts need to adapt their analytical processes after each API release. This gets more challenging when performing an integrated or historical analysis. To cope with such complexity, in this paper, we present the Big Data Integration ontology, the core construct to govern the data integration process under schema evolution by systematically annotating it with in…
Large organizations are seeking to create new architectures and scalable platforms to effectively handle data management challenges due to the explosive nature of data rarely seen in the past. These data management challenges are largely posed by the availability of streaming data at high velocity from various sources in multiple formats. The changes in data paradigm have led to the emergence of new data analytics and management architecture. This paper focuses on storing high volume, velocity and variety data in the raw formats in a data storage architecture called a data lake. First, we pres…
Many astronomy data centres still work on filesystems. Industry has moved on; current practice in computing infrastructure is to achieve Big Data scalability using object stores rather than POSIX file systems. This presents us with opportunities for portability and reuse of software underlying processing and archive systems but it also causes problems for legacy implementations in current data centers.
One of the great challenges the information society faces is dealing with the huge amount of information generated and handled daily on the Internet. Today, progress in Big data proposals attempts to solve this problem, but there are certain limitations to information search and retrieval due basically to the large volumes handled the heterogeneity of the information, and its dispersion among a multitude of sources. In this article, a formal framework is defined to facilitate the design and development of an environmental management information system, which works with a heterogeneous and larg…
Delivering effective data analytics is of crucial importance to the interpretation of the multitude of biological datasets currently generated by an ever increasing number of high throughput techniques. Logic programming has much to offer in this area. Here, we detail advances that highlight two of the strengths of logical formalisms in developing data analytic solutions in biological settings: access to large relational databases and building analytical pipelines collecting graph information from multiple sources. We present significant advances on the bio_db package which serves biological d…
Big Data technology is described. Big data is a popular term used to describe the exponential growth and availability of data, both structured and unstructured. There is constructed dataspace architecture. Dataspace has focused solely - and passionately - on providing unparalleled expertise in business intelligence and data warehousing strategy and implementation. Dataspaces are an abstraction in data management that aims to overcome some of the problems encountered in data integration system. In our case it is block vector for heterogeneous data representation. Traditionally, data integration…
