The aim of this article is to present an overview of the major families of state-of-the-art index and materialized view selection methods, and to discuss the issues and future trends in data warehouse performance optimization. We particularly focus on data mining-based heuristics we developed to reduce the selection problem complexity and target the most pertinent candidate indexes and materialized views.
A data warehouse is a large data repository for the purpose of analysis and decision making in organizations. To improve the query performance and to get fast access to the data, data is stored as materialized views (MV) in the data warehouse. When data at source gets updated, the materialized views also need to be updated. In this paper, we focus on the problem of maintenance of these materialized views and address the issue of finding such auxiliary views (AV) that together with the materialized views make the data self-maintainable and take minimal space. We propose an algorithm that uses k…
Data warehousing is an essential element of decision support systems. It aims at enabling the user knowledge to make better and faster daily business decisions. To improve this decision support system and to give more and more relevant information to the user, the need to integrate user's profiles into the data warehouse process becomes crucial. In this paper, we propose to exploit users' preferences as a basis for adapting OLAP (On-Line Analytical Processing) queries to the user. For this, we present a user profile-driven data warehouse approach that allows dening user's profile composed by h…
Data warehouse performance is usually achieved through physical data structures such as indexes or materialized views. In this context, cost models can help select a relevant set ofsuch performance optimization structures. Nevertheless, selection becomes more complex in the cloud. The criterion to optimize is indeed at least two-dimensional, with monetary cost balancing overall query response time. This paper introduces new cost models that fit into the pay-as-you-go paradigm of cloud computing. Based on these cost models, an optimization problem is defined to discover, among candidate views, …
XML data warehouses form an interesting basis for decision-support applications that exploit complex data. However, native-XML database management systems (DBMSs) currently bear limited performances and it is necessary to research for ways to optimize them. In this chapter, we present two such techniques. First, we propose a join index that is specifically adapted to the multidimensional architecture of XML warehouses. It eliminates join operations while preserving the information contained in the original warehouse. Second, we present a strategy for selecting XML materialized views by cluster…
With the development of decision systems and specially data warehouses, the visibility of the data warehouse design before its creation has become essential, and that because of data warehouse importance as considered as the unique data source giving meaning to the decision. In a decision system the proper functioning of a data warehouse resides in the smooth running of the middleware tools ETC step one hand, and the restitution step through the data mining, reporting solutions, dashboards... etc other. The large volume of data that passes through these stages require an optimal design for a h…
Data warehouses are overwhelmingly built through a bottom-up process, which starts with the identification of sources, continues with the extraction and transformation of data from these sources, and then loads the data into a set of data marts according to desired multidimensional relational schemas. End user business intelligence tools are added on top of the materialized multidimensional schemas to drive decision making in an organization. Unfortunately, this bottom-up approach is costly both in terms of the skilled users needed and the sheer size of the warehouses. This paper proposes a to…
With the need for flexible and on-demand decision support, Dynamic Data Warehouses (DDW) provide benefits over traditional data warehouses due to their dynamic characteristics in structuring and access mechanism. A DDW is a data framework that accommodates data source changes easily to allow seamless querying to users. Materialized Views (MV) are proven to be an effective methodology to enhance the process of retrieving data from a DDW as results are pre-computed and stored in it. However, due to the static nature of materialized views, the level of dynamicity that can be provided at the MV ac…
Materialized views (MVs), stored pre-computed results, are widely used to facilitate fast queries on large datasets. When new records arrive at a high rate, it is infeasible to continuously update (maintain) MVs and a common solution is to defer maintenance by batching updates together. Between batches the MVs become increasingly stale with incorrect, missing, and superfluous rows leading to increasingly inaccurate query results. We propose Stale View Cleaning (SVC) which addresses this problem from a data cleaning perspective. In SVC, we efficiently clean a sample of rows from a stale MV, and…
Decision systems deal with a large volume of data stored in new databases called data warehouses. Data warehouses are typically modeled by a star schema that conventionally presents a central fact table and a set of dimension tables. The corresponding queries for this type of model are therefore very complex. In order to reduce the cost of executing complex queries, which contain very expensive joins, the solution envisaged would be to guarantee a good physical design of the data warehouses. Binary join indexes are very suitable to reduce the cost of executing these joins. In this work, we pro…
Presently, large enterprises rely on database systems to manage their data and information. These databases are useful for conducting daily business transactions. However, the tight competition in the marketplace has led to the concept of data mining in which data are analyzed to derive effective business strategies and discover better ways in carrying out business. In order to perform data mining, regular databases must be converted into what so called informational databases also known as data warehouse. This paper presents a design model for building data warehouse for a typical university …
The article discusses the approach for evaluating the innovation index of the products and technologies. The evaluation results can be used to create a warehouse of the object descriptions with significant innovation potential. The model of innovation index computation is based on the concepts of novelty, relevance, and implementability of the object. Formal definitions of these indicators are given and a methodology for their calculation are described. The fuzzy methods to coprocess (incomplete) data from numerous sources and to obtain probabilistic innovation assessments are used. The experi…
This paper presents a family of algorithms for fast subset filtering within ordered sets of integers representing composite keys. Applications include significant acceleration of (ad-hoc) analytic queries against a data warehouse without any additional indexing. The algorithms work for point, range and set restrictions on multiple attributes, in any combination, and are inherently multidimensional. The main idea consists in intelligent combination of sequential crawling with jumps over large portions of irrelevant keys. The way to combine them is adaptive to characteristics of the underlying d…
Filtering data based on predicates is one of the most fundamental operations for any modern data warehouse. Techniques to accelerate the execution of filter expressions include clustered indexes, specialized sort orders (e.g., Z-order), multi-dimensional indexes, and, for high selectivity queries, secondary indexes. However, these schemes are hard to tune and their performance is inconsistent. Recent work on learned multi-dimensional indexes has introduced the idea of automatically optimizing an index for a particular dataset and workload. However, the performance of that work suffers in the p…
