We study how software engineers design and evolve their domain model when building applications against NoSQL data stores. Specifically, we target Java projects that use object-NoSQL mappers to interface with schema-free NoSQL data stores. Given the source code of ten real-world database applications, we extract the implicit NoSQL database schema. We capture the sizes of the schemas, and investigate whether the schema is denormalized, as is recommended practice in data modeling for NoSQL data stores. Further, we analyze the entire project history, and with it, the evolution history of the NoSQL database schema. In doing so, we conduct the so far largest empirical study on NoSQL schema design and evolution.
In this paper, we propose a schema optimization method for time-dependent workloads for NoSQL databases. In our proposed method, we migrate schema according to changing workloads, and the estimated cost of execution and migration are formulated and minimized as a single integer linear programming problem. Furthermore, we propose a method to reduce the number of optimization candidates by iterating over the time dimension abstraction and optimizing the workload while updating constraints.
Schema evolution is a crucial aspect in database management. The proposed taxonomies of schema changes have neglected the set of operations that involves relationships between entity types: aggregation and references, as well as the possible existence of structural variations for schema types, as most of NoSQL systems are schemaless. The distinction between entity types and relationship types, which is typical of graph schemas, is also not taken into account in the published works. Moreover, NoSQL schema evolution poses the challenge of having different data models, and no standard specificati…
NoSQL data stores are commonly schema-less, providing no means for globally defining or managing the schema. While this offers great flexibility in early stages of application development, developers soon can experience the heavy burden of dealing with increasingly heterogeneous data. This paper targets schema evolution for NoSQL data stores, the complex task of adapting and changing the implicit structure of the data stored. We discuss the recommendations of the developer community on handling schema changes, and introduce a simple, declarative schema evolution language. With our language, so…
The demanding requirements of the new Big Data intensive era raised the need for flexible storage systems capable of handling huge volumes of unstructured data and of tackling the challenges that traditional databases were facing. NoSQL Databases, in their heterogeneity, are a powerful and diverse set of databases tailored to specific industrial and business needs. However, the lack of theoretical background creates a lack of consensus even among experts about many NoSQL concepts, leading to ambiguity and confusion. In this paper, we present a survey of NoSQL databases and their classification…
In this document, I present the main notions of NoSQL databases and compare four selected products (Riak, MongoDB, Cassandra, Neo4J) according to their capabilities with respect to consistency, availability, and partition tolerance, as well as performance. I also propose a few criteria for selecting the right tool for the right situation.
This paper explores methodologies, advantages and challenges related to the use of the Information Centric Network technology for developing NoSQL distributed databases, which are expected to play a central role in the forthcoming IoT and BigData era. ICN services make possible to simplify the development of the database software, improve performance, and provide data-level access control. We use our findings for devising a NoSQL spatio-temporal database, named OpenGeoBase, and evaluate its performance with a real data set related to Intelligent Transport System applications.
Increasing resource demands require relational databases to scale. While relational databases are well suited for vertical scaling, specialized hardware can be expensive. Conversely, emerging NewSQL and NoSQL data stores are designed to scale horizontally. NewSQL databases provide ACID transaction support; however, joins are limited to the partition keys, resulting in restricted query expressiveness. On the other hand, NoSQL databases are designed to scale out linearly on commodity hardware; however, they are limited by slow join performance. Hence, we consider if the NoSQL join performance ca…
NoSQL databases have become an important component of many big data and real-time web applications. Their distributed nature and scalability make them an ideal data storage repository for a variety of use cases. While NoSQL databases are delivered with a default ''off-the-shelf'' configuration, they offer configuration settings to adjust a database's behavior and performance to a specific use case and environment. The abundance and oftentimes imperceptible inter-dependencies of configuration settings make it difficult to optimize and performance-tune a NoSQL system. There is no one-size-fits-a…
Digital world is growing very fast and become more complex in the volume (terabyte to petabyte), variety (structured and un-structured and hybrid), velocity (high speed in growth) in nature. This refers to as Big Data that is a global phenomenon. This is typically considered to be a data collection that has grown so large it can not be effectively managed or exploited using conventional data management tools: e.g., classic relational database management systems (RDBMS) or conventional search engines. To handle this problem, traditional RDBMS are complemented by specifically designed a rich set…
We have created a cloud-based service that allows the end users to run tests on multiple different databases to find which databases are most suitable for their project. From our research, we could not find another application that enables the user to test several databases to gauge the difference between them. This application allows the user to choose which type of test to perform and which databases to target. The application also displays the results of different tests that were run by other users previously. There is also a map to show the location where all the tests are run to give the …
NoSQL databases like Redis, Cassandra, and MongoDB are increasingly popular because they are flexible, lightweight, and easy to work with. Applications that use these databases will evolve over time, sometimes necessitating (or preferring) a change to the format or organization of the data. The problem we address in this paper is: How can we support the evolution of high-availability applications and their NoSQL data online, without excessive delays or interruptions, even in the presence of backward-incompatible data format changes? We present KVolve, an extension to the popular Redis NoSQL da…
We propose a new approach of NoSQL database index selection. For different workloads, we select different indexes and their different parameters to optimize the database performance. The approach builds a deep reinforcement learning model to select an optimal index for a given fixed workload and adapts to a changing workload. Experimental results show that, Deep Reinforcement Learning Index Selection Approach (DRLISA) has improved performance to varying degrees according to traditional single index structures.
The digital transformation of companies has led to the evolution of databases towards Big Data. Our work is part of this context and concerns more particularly the mechanisms to extract datasets stored in a Data Lake and to store the data in a Data Warehouse. The latter will allow, in a second time, decisional analysis. In this paper, we present the extraction mechanism limited to relational databases. To automate this process, we used the Model Driven Architecture (MDA), which offers a formalized environment for schema transformation. From the physical schemas describing a Data Lake, we propo…
With the continuous development of NoSQL databases, more and more developers choose to use semi-structured data for development and data management, which puts forward requirements for schema management of semi-structured data stored in NoSQL databases. Schema extraction plays an important role in understanding schemas, optimizing queries, and validating data consistency. Therefore, in this survey we investigate structural methods based on tree and graph and statistical methods based on distributed architecture and machine learning to extract schemas. The schemas obtained by the structural met…
During the development of NoSQL-backed software, the data model evolves naturally alongside the application code. Especially in agile development, new application releases are deployed frequently causing schema changes. Eventually, decisions have to be made regarding the migration of versioned legacy data which is persisted in the cloud-hosted production database. We solve this schema evolution problem and present the results of near-exhaustive calculations by means of which software project stakeholders can manage the operative costs for data model evolution and adapt their software release s…
During the last few years, the explosion of Big Data has prompted cloud infrastructures to provide cloud-based database services as cost effective, efficient and scalable solutions to store and process large volume of data. Hence, NoSQL databases became more and more popular because of their inherent features of better performance and high scalability compared to other relational databases. However, with this deployment architecture where the information is stored in a public cloud, protection against the sensitive data is still being a major concern. Since the data owner does not have the ful…
In recent years, the increased need to house and process large volumes of data has prompted the need for distributed storage and querying systems. The growth of machine-readable RDF triples has prompted both industry and academia to develop new database systems, called NoSQL, with characteristics that differ from classical databases. Many of these systems compromise ACID properties for increased horizontal scalability and data availability. This thesis concerns the development and evaluation of a NoSQL triplestore. Triplestores are database management systems central to emerging technologies s…
Process-Aware Information Systems (PAIS) is an IT system that support business processes and generate large amounts of event logs from the execution of business processes. An event log is represented as a tuple of CaseID, Timestamp, Activity and Actor. Process Mining is a new and emerging field that aims at analyzing the event logs to discover, enhance and improve business processes and check conformance between run time and design time business processes. The large volume of event logs generated are stored in the databases. Relational databases perform well for a certain class of applications…
In recent years, the need to use NoSQL systems to store and exploit big data has been steadily increasing. Most of these systems are characterized by the property "schema less" which means absence of the data model when creating a database. This property brings an undeniable flexibility by allowing the evolution of the model during the exploitation of the base. However, the expression of queries requires a precise knowledge of this model. In this paper, we propose an incremental process to extract the model while operating the document-oriented NoSQL database. To do this, we use the Model Driv…
Context: The efficient processing of Big Data is a challenging task for SQL and NoSQL Databases, where competent software architecture plays a vital role. The SQL Databases are designed for structuring data and supporting vertical scalability. In contrast, horizontal scalability is backed by NoSQL Databases and can process sizeable unstructured Data efficiently. One can choose the right paradigm according to the organisation's needs; however, making the correct choice can often be challenging. The SQL and NoSQL Databases follow different architectures. Also, the mixed model is followed by each…
Various data consistency levels have an important part in the integrity of data and also affect performance especially the data that is replicated many times across or over the cluster. Based on BASE and the theorem of CAP tradeoffs, most systems of NoSQL have more relaxed consistency guarantees than another kind of databases which implement ACID. Most systems of NoSQL gave different methods to adjust a required level of consistency to ensure the minimal numbering of the replicas accepted in each operation. Simulations are always depending on a simplified model and ignore many details and fact…
One of the key advances in resolving the big-data problem has been the emergence of an alternative database technology. Today, classic RDBMS are complemented by a rich set of alternative Data Management Systems (DMS) specially designed to handle the volume, variety, velocity and variability ofBig Data collections; these DMS include NoSQL, NewSQL and Search-based systems. NewSQL is a class of modern relational database management systems (RDBMS) that provide the same scalable performance of NoSQL systems for online transaction processing (OLTP) read-write workloads while still maintaining the A…
The success of SQL, NoSQL, and NewSQL databases is a reflection of their ability to provide significant functionality and performance benefits for specific domains, such as financial transactions, internet search, and data analysis. The BigDAWG polystore seeks to provide a mechanism to allow applications to transparently achieve the benefits of diverse databases while insulating applications from the details of these databases. Associative arrays provide a common approach to the mathematics found in different databases: sets (SQL), graphs (NoSQL), and matrices (NewSQL). This work presents the …
The World Wide Web infrastructure together with its more than 2 billion users enables to store information at a rate that has never been achieved before. This is mainly due to the will of storing almost all end-user interactions performed on some web applications. In order to reply to scalability and availability constraints, many web companies involved in this process recently started to design their own data management systems. Many of them are referred to as NOSQL databases, standing for 'Not only SQL'. With their wide adoption emerges new needs and data integration is one of them. In this …
Big data and the Internet of Things era continue to challenge computational systems. Several technology solutions such as NoSQL databases have been developed to deal with this challenge. In order to generate meaningful results from large datasets, analysts often use a graph representation which provides an intuitive way to work with the data. Graph vertices can represent users and events, and edges can represent the relationship between vertices. Graph algorithms are used to extract meaningful information from these very large graphs. At MIT, the Graphulo initiative is an effort to perform gra…
Google BigTable's scale-out design for distributed key-value storage inspired a generation of NoSQL databases. Recently the NewSQL paradigm emerged in response to analytic workloads that demand distributed computation local to data storage. Many such analytics take the form of graph algorithms, a trend that motivated the GraphBLAS initiative to standardize a set of matrix math kernels for building graph algorithms. In this article we show how it is possible to implement the GraphBLAS kernels in a BigTable database by presenting the design of Graphulo, a library for executing graph algorithms i…
Epidemiologist, Scientists, Statisticians, Historians, Data engineers and Data scientists are working on finding descriptive models and theories to explain COVID-19 expansion phenomena or on building analytics predictive models for learning the apex of COVID-19 confimed cases, recovered cases, and deaths evolution curves. In CRISP-DM life cycle, 75% of time is consumed only by data preparation phase causing lot of pressions and stress on scientists and data scientists building machine learning models. This paper aims to help reducing data preparation efforts by presenting detailed schemas desi…
Nowadays, there are billions interconnected devices forming Cyber-Physical Systems, Internet of Things (IoT) and Industrial Internet of Things (IIoT) ecosystems. With an increasing number of devices and systems in use, amount and the value of data, the risks of security breaches increase. One of these risks is posed by open data sources, by which are meant databases, which are not properly protected. These poorly protected databases are accessible to external actors, which poses a serious risk to the data holder and the results of data-related activities such as analysis, forecasting, monitori…
Ascertaining the feasibility of independent falsification or repetition of published results is vital to the scientific process, and replication or reproduction experiments are routinely performed in many disciplines. Unfortunately, such studies are only scarcely available in database research, with few papers dedicated to re-evaluating published results. In this paper, we conduct a case study on replicating and reproducing a study on schema evolution in embedded databases. We obtain exact results for one out of four database applications studied, and come close in two further cases. By report…
COVID-19's rapid global spread has driven innovative tools for Big Data Analytics. These have guided organizations in all fields of the health industry to track and minimized the effects of virus. Researchers are required to detect coronaviruses through artificial intelligence, machine learning, and natural language processing, and to gain a complete understanding of the disease. COVID-19 takes place in different countries in the world, with which only big data application and the work of NOSQL databases are suitable. There is a great number of platforms used for processing NOSQL Databases mod…
One of the challenging problems in the multidatabase systems is to find the most viable solution to the problem of interoperability of distributed heterogeneous autonomous local component databases. This has resulted in the creation of a global schema over set of these local component database schemas to provide a uniform representation of local schemas. The aim of this paper is to use object oriented approach to integrate schemas of distributed heterogeneous autonomous local component database schemas into a global schema. The resulting global schema provides a uniform interface and high leve…
According to strong demands for rapid and reliable software delivery, co-existing database schema versions with multiple application versions are reality to contribute them. Current database management systems do not support co-existing schema versions in one database. Although a design of co-existing schema based on updatable view tables was previously proposed, its flexibility is limited due to pre-defined several restrictions to achieve data synchronization among schemas and handling independent unsynchronized data in each schema. In this preliminary report, we present a new approach for co…
Modern database applications often change their schemas to keep up with the changing requirements. However, support for online and transactional schema evolution remains challenging in existing database systems. Specifically, prior work often takes ad hoc approaches to schema evolution with 'patches' applied to existing systems, leading to many corner cases and often incomplete functionality. Applications therefore often have to carefully schedule downtimes for schema changes, sacrificing availability. This paper presents Tesseract, a new approach to online and transactional schema evolution w…
We introduce end-to-end support of co-existing schema versions within one database. While it is state of the art to run multiple versions of a continuously developed application concurrently, it is hard to do the same for databases. In order to keep multiple co-existing schema versions alive; which are all accessing the same data set; developers usually employ handwritten delta code (e.g. views and triggers in SQL). This delta code is hard to write and hard to maintain: if a database administrator decides to adapt the physical table schema, all handwritten delta code needs to be adapted as wel…
In this article we discuss an approach to database optimisation in which a conceptual schema is optimised by applying a sequence of transformations. By performing these optimisations on the conceptual schema, a large part of the database optimisation can be done before actually sliding down the software development waterfall. When optimising schemas, one would like to preserve some level of equivalence between the schemas before and after a transformation. We distinguish between two classes of equivalence, one based on the mathematical semantics of the conceptual schemas, and one on conceptual…
Despite the maturity of commercial graph databases, little consensus has been reached so far on the standardization of data definition languages (DDLs) for property graphs (PG). The discussion on the characteristics of PG schemas is ongoing in many standardization and community groups. Although some basic aspects of a schema are already present in Neo4j 3.5, like in most commercial graph databases, full support is missing allowing to constraint property graphs with more or less flexibility. In this paper, we focus on two different perspectives from which a PG schema should be considered, as be…
In the current context of Big Data, a multitude of new NoSQL solutions for storing, managing, and extracting information and patterns from semi-structured data have been proposed and implemented. These solutions were developed to relieve the issue of rigid data structures present in relational databases, by introducing semi-structured and flexible schema design. As current data generated by different sources and devices, especially from IoT sensors and actuators, use either XML or JSON format, depending on the application, database technologies that store and query semi-structured data in XML …
NoSQL databases are becoming increasingly popular as more developers seek new ways for storing information. The popularity of these databases has risen due to their flexibility and scalability needed in domains like Big Data and Cloud Computing. This paper examines asynchronous replication, one of the key features for a scalable and flexible system. Three of the most popular Document-Oriented Databases, MongoDB, CouchDB, and Couchbase, are examined. For testing, the execution time for CRUD operations for a single database instance and for a distributed environment with two nodes is taken into …
