Distributed data processing platforms (e.g., Hadoop, Spark, and Flink) are widely used to distribute the storage and processing of data among computing nodes of a cloud. The centralization of cloud resources has given birth to edge computing, which enables the processing of data closer to the data source instead of sending it to the cloud. However, due to resource constraints such as energy limitations, edge computing cannot be used for deploying all kinds of applications. Therefore, tasks are offloaded from an edge device to the more resourceful cloud. Previous research has evaluated the energy consumption of the distributed data processing platforms in the isolated cloud and edge environments. However, there is a paucity of research on evaluating the energy consumption of these platforms in an integrated edge-cloud environment, where tasks are offloaded from a resource-constraint device to a resource-rich device. Therefore, in this paper, we first present a framework for the energy-aware evaluation of the distributed data processing platforms. We then leverage the proposed framework to evaluate the energy consumption of the three most widely used platforms (i.e., Hadoop, Spark, and Flink) in an integrated edge-cloud environment consisting of Raspberry Pi, edge node, edge server node, private cloud, and public cloud. Our evaluation reveals that (i) Flink is most energy-efficient followed by Spark and Hadoop is found least energy-efficient (ii) offloading tasks from resource-constraint to resource-rich devices reduces energy consumption by 55.2%, and (iii) bandwidth and distance between client and server are found key factors impacting the energy consumption.
This paper made a short review of Cloud Computing and Big Data, and discussed the portability of general data mining algorithms to Cloud Computing platform. It revealed the Cloud Computing platform based on Map-Reduce cannot solve all the Big Data and data mining problems. Transplanting the general data mining algorithms to the real-time Cloud Computing platform will be one of the research focuses in Cloud Computing and Big Data.
With the advantages that cloud computing offers in terms of platform as a service, software as a service, and infrastructure as a service, data engineers and data scientists are able to leverage cloud computing for their ETL/ELT (extract, transform and load) and ML (machine learning) requirements and deployments. The proposed framework for the comparative review of cloud computing platforms for data science workflows uses an amalgamation of the analytical hierarchy process, Saaty's fundamental scale of absolute numbers, and a selection of relevant evaluation criteria (namely: automation, error…
Cloud computing is rapidly emerging as a new paradigm for delivering IT services as utlity-oriented services on subscription-basis. The rapid development of applications and their deployment in Cloud computing environments in efficient manner is a complex task. In this article, we give a brief introduction to Cloud computing technology and Platform as a Service, we examine the offerings in this category, and provide the basis for helping readers to understand basic application platform opportunities in Cloud by technologies such as Microsoft Azure, Sales Force, Google App, and Aneka for Cloud …
We introduce Cloud4IoT, a platform offering automatic deployment, orchestration and dynamic configuration of IoT support software components and data-intensive applications for data processing and analytics, thus enabling plug-and-play integration of new sensor objects and dynamic workload scalability. Cloud4IoT enables the concept of Infrastructure as Code in the IoT context: it empowers IoT operations with the flexibility and elasticity of Cloud services. Furthermore it shifts traditionally centralized Cloud architectures towards a more distributed and decentralized computation paradigm, as …
Cloud Computing is becoming a viable computing solution for services oriented computing. Several open-source cloud solutions are available to these supports. Open-source software stacks offer a huge amount of customizability without huge licensing fees. As a result, open source software are widely used for designing cloud, and private clouds are being built increasingly in the open source way. Numerous contributions have been made by the open-source community related to private-IaaS-cloud. OpenNebula - a cloud platform is one of the popular private cloud management software. However, little ha…
Cloud services have been widely employed in IT industry and scientific research. By using Cloud services users can move computing tasks and data away from local computers to remote datacenters. By accessing Internet-based services over lightweight and mobile devices, users deploy diversified Cloud applications on powerful machines. The key drivers towards this paradigm for the scientific computing field include the substantial computing capacity, on-demand provisioning and cross-platform interoperability. To fully harness the Cloud services for scientific computing, however, we need to design …
This paper describes the use of a distributed cloud computing system for high-throughput computing (HTC) scientific applications. The distributed cloud computing system is composed of a number of separate Infrastructure-as-a-Service (IaaS) clouds that are utilized in a unified infrastructure. The distributed cloud has been in production-quality operation for two years with approximately 500,000 completed jobs where a typical workload has 500 simultaneous embarrassingly-parallel jobs that run for approximately 12 hours. We review the design and implementation of the system which is based on pre…
Aneka is an Application Platform-as-a-Service (Aneka PaaS) for Cloud Computing. It acts as a framework for building customized applications and deploying them on either public or private Clouds. One of the key features of Aneka is its support for provisioning resources on different public Cloud providers such as Amazon EC2, Windows Azure and GoGrid. In this chapter, we will present Aneka platform and its integration with one of the public Cloud infrastructures, Windows Azure, which enables the usage of Windows Azure Compute Service as a resource provider of Aneka PaaS. The integration of the t…
The availability of Infrastructure-as-a-Service (IaaS) computing clouds gives researchers access to a large set of new resources for running complex scientific applications. However, exploiting cloud resources for large numbers of jobs requires significant effort and expertise. In order to make it simple and transparent for researchers to deploy their applications, we have developed a virtual machine resource manager (Cloud Scheduler) for distributed compute clouds. Cloud Scheduler boots and manages the user-customized virtual machines in response to a user's job submission. We describe the mo…
A method for evaluating the relative performance of local, cloud and combined processing of divisible (i.e. partitionable) data loads is presented. It is shown how to do this in the context of Amdahl's law. A single level (star) network operating under each of three fundamental scheduling policies is used as an example. Applications include mobile computing, cloud computing and signature searching.
Bag-of-Distributed-Tasks (BoDT) application is the collection of identical and independent tasks each of which requires a piece of input data located around the world. As a result, Cloud computing offers an ef- fective way to execute BoT application as it not only consists of multiple geographically distributed data centres but also allows a user to pay for what she actually uses only. In this paper, BoDT on the Cloud using virtually unlimited cloud resources. A heuristic algorithm is proposed to find an execution plan that takes budget constraints into account. Compared with other approaches,…
A Hybrid cloud is an integration of resources between private and public clouds. It enables users to horizontally scale their on-premises infrastructure up to public clouds in order to improve performance and cut up-front investment cost. This model of applications deployment is called cloud bursting that allows data-intensive applications especially distributed database systems to have the benefit of both private and public clouds. In this work, we present an automated implementation of a hybrid cloud using (i) a robust and zero-cost Linux-based VPN to make a secure connection between private…
The IP Multimedia Subsystem (IMS) defined by the 3GPP has been mainly developed and deployed by telephony vendors on vendor-specific hardware. Recent advances in Network Function Virtualisation (NFV) technology paved the way for virtualized hardware and telephony function elasticity. As such, Telecom vendors have started to embrace the cloud as a deployment platform, usually selecting a privileged virtualization platform. Operators would like to deploy telecom functionality on their already existing IT cloud platforms. Achieving such flexibility would require the telecom vendors to adopt a sof…
Energy efficiency is a crucial factor in the well-being of our planet. In parallel, Machine Learning (ML) plays an instrumental role in automating our lives and creating convenient workflows for enhancing behavior. So, analyzing energy behavior can help understand weak points and lay the path towards better interventions. Moving towards higher performance, cloud platforms can assist researchers in conducting classification trials that need high computational power. Under the larger umbrella of the Consumer Engagement Towards Energy Saving Behavior by means of Exploiting Micro Moments and Mobil…
Workflow management systems (WMS) support the composition and deployment of workflow-oriented applications in distributed computing environments. They hide the complexity of managing large-scale applications, which includes the controlling data pipelining between tasks, ensuring the application's execution, and orchestrating the distributed computational resources to get a reasonable processing time. With the increasing trends of scientific workflow adoption, the demand to deploy them using a third-party service begins to increase. Workflow-as-a-service (WaaS) is a term representing the platfo…
We present the architecture of a cloud native version of IBM Streams, with Kubernetes as our target platform. Streams is a general purpose streaming system with its own platform for managing applications and the compute clusters that execute those applications. Cloud native Streams replaces that platform with Kubernetes. By using Kubernetes as its platform, Streams is able to offload job management, life cycle tracking, address translation, fault tolerance and scheduling. This offloading is possible because we define custom resources that natively integrate into Kubernetes, allowing Streams to…
In this paper, we present the ADMIRE architecture; a new framework for developing novel and innovative data mining techniques to deal with very large and distributed heterogeneous datasets in both commercial and academic applications. The main ADMIRE components are detailed as well as its interfaces allowing the user to efficiently develop and implement their data mining applications techniques on a Grid platform such as Globus ToolKit, DGET, etc.
It is estimated that data centers constitute 1.5% of global electricity usage. At the same time, to serve increasing user requirements, modern cloud providers are operating multiple geographically distributed data centers. Distributed data center infrastructure changes the rules of cloud control, as energy costs depend on current regional electricity prices and temperatures that we call geotemporal inputs. Furthermore, pricing policies at which cloud providers can offer computational resources depend on the quality of service (QoS). With such pricing schemes and the increasing energy costs in …
The explosive increase in volume, velocity, variety, and veracity of data generated by distributed and heterogeneous nodes such as IoT and other devices, continuously challenge the state of art in big data processing platforms and mining techniques. Consequently, it reveals an urgent need to address the ever-growing gap between this expected exascale data generation and the extraction of insights from these data. To address this need, this paper proposes Stream to Cloud & Edge (S2CE), a first of its kind, optimized, multi-cloud and edge orchestrator, easily configurable, scalable, and extensib…
Edge computing is an emerging technology which places computing at the edge of the network to provide an ultra-low latency. Computation offloading, a paradigm that migrates computing from mobile devices to remote servers, can now use the power of edge computing by offloading computation to cloudlets in edge-clouds. However, the task scheduling of computation offloading in edge-clouds faces a two-fold challenge. First, as cloudlets are geographically distributed, it is difficult for each cloudlet to perform load balancing without centralized control. Second, as tasks of computation offloading h…
Internet of Things typically involves a significant number of smart sensors sensing information from the environment and sharing it to a cloud service for processing. Various architectural abstractions, such as Fog and Edge computing, have been proposed to localize some of the processing near the sensors and away from the central cloud servers. In this paper, we propose Edge-Fog Cloud which distributes task processing on the participating cloud resources in the network. We develop the Least Processing Cost First (LPCF) method for assigning the processing tasks to nodes which provide the optima…
The paper suggests a new approach based on blockchain technologies and smart contracts to creation of a distributed system for managing provenance metadata, as well as access rights to data in distributed storages, which is fault-tolerant, safe and secure from the point of view of preservation of metadata records from accidental or intentional distortions. The implementation of the proposed approach is based on the permissioned blockchains and on the Hyperledger Fabric blockchain platform in conjunction with Hyperledger Composer.
Cloud computing is the way by which we connect to servers, large systems into a distributed secure manner without worrying about local memory limits. Here this paper we proposed a Novel Distributed Database Architectural Model for Mobile Cloud Computing (NDDAMMCC). Due to the exponential growth of wireless technologies and internet which are following Nielsen's Law of Internet Bandwidth, we are in the new era of cloud computing. In the recent technological era, smart mobile devices play a big role in all sort of day-by-day human needs. The applicability is so huge that the number of apps insta…
Distributed data processing frameworks (e.g., Hadoop, Spark, and Flink) are widely used to distribute data among computing nodes of a cloud. Recently, there have been increasing efforts aimed at evaluating the performance of distributed data processing frameworks hosted in private and public clouds. However, there is a paucity of research on evaluating the performance of these frameworks hosted in a hybrid cloud, which is an emerging cloud model that integrates private and public clouds to use the best of both worlds. Therefore, in this paper, we evaluate the performance of Hadoop, Spark, and …
The ICT (Information Communication Technologies) ecosystem is estimated to be responsible, as of today, for 10% of the total worldwide energy demand - equivalent to the combined energy production of Germany and Japan. Cloud storage, mainly operated through large and densely-packed data centers, constitutes a non-negligible part of it. However, since the cloud is a fast-inflating market and the energy-efficiency of data centers is mostly an insensitive issue for the collectivity, its carbon footprint shows no signs of slowing down. In this paper, we analyze a novel paradigm for cloud storage (i…
Distributed fog and edge applications communicate over unreliable networks and are subject to high communication delays. This makes using existing distributed coordination technologies from cloud applications infeasible, as they are built on the assumption of a highly reliable, low-latency datacenter network to achieve strict consistency with low overheads. To help implement configuration and state management for fog platforms and applications, we propose a novel decentralized approach that lets systems specify coordination strategies and membership for different sets of coordination data.
In recent years, artificial intelligence (AI) has aroused much attention among both industrial and academic areas. However, building and maintaining efficient AI systems are quite difficult for many small business companies and researchers if they are not familiar with machine learning and AI. In this paper, we first evaluate the difficulties and challenges in building AI systems. Then an cloud platform termed XCloud, which provides several common AI services in form of RESTful APIs, is constructed. Technical details are discussed in Section 2. This project is released as open-source software …
The global economic recession and the shrinking budget of IT projects have led to the need of development of integrated information systems at a lower cost. Today, the emerging phenomenon of cloud computing aims at transforming the traditional way of computing by providing both software applications and hardware resources as a service. With the rapid evolution of Information Communication Technology (ICT) governments, organizations and businesses are looking for solutions to improve their services and integrate their IT infrastructures. In recent years advanced technologies such as SOA and Clo…
We investigate a hierarchically organized cloud infrastructure and compare distributed hierarchical control based on resource monitoring with market mechanisms for resource management. The latter do not require a model of the system, incur a low overhead, are robust, and satisfy several other desiderates of autonomic computing. We introduce several performance measures and report on simulation studies which show that a straightforward bidding scheme supports an effective admission control mechanism, while reducing the communication complexity by several orders of magnitude and also increasing …
The paper illustrates how we built a federated cloud computing platform dedicated to the Italian research community. Building a cloud platform is a daunting task, that requires coordinating the deployment of many services, interrelated and dependent on each other. Provisioning, servicing and maintaining the platform must be automated. For our deployment, we chose a declarative modeling tool, that allows describing the parts that compose the system and their relations of supplier/consumer of specific interfaces. The tool arranges the steps to bring the deployment to convergence by transforming …
Cloud Computing researches involve a tremendous amount of entities such as users, applications, and virtual machines. Due to the limited access and often variable availability of such resources, researchers have their prototypes tested against the simulation environments, opposed to the real cloud environments. Existing cloud simulation environments such as CloudSim and EmuSim are executed sequentially, where a more advanced cloud simulation tool could be created extending them, leveraging the latest technologies as well as the availability of multi-core computers and the clusters in the resea…
At present there are a number of barriers to creating an energy efficient workload scheduler for a Private Cloud based data center. Firstly, the relationship between different workloads and power consumption must be investigated. Secondly, current hardware-based solutions to providing energy usage statistics are unsuitable in warehouse scale data centers where low cost and scalability are desirable properties. In this paper we discuss the effect of different workloads on server power consumption in a Private Cloud platform. We display a noticeable difference in energy consumption when servers …
Bag of Distributed Tasks (BoDT) can benefit from decentralised execution on the Cloud. However, there is a trade-off between the performance that can be achieved by employing a large number of Cloud VMs for the tasks and the monetary constraints that are often placed by a user. The research reported in this paper is motivated towards investigating this trade-off so that an optimal plan for deploying BoDT applications on the cloud can be generated. A heuristic algorithm, which considers the user's preference of performance and cost is proposed and implemented. The feasibility of the algorithm i…
According to the pay-per-use model adopted in clouds, the more the resources consumed by an application running in a cloud computing environment, the greater the amount of money the owner of the corresponding application will be charged. Therefore, applying intelligent solutions to minimize the resource consumption is of great importance. Because centralized solutions are deemed unsuitable for large-distributed systems or large-scale applications, we propose a fully distributed algorithm (called DRA) to overcome the scalability issues. Specifically, DRA migrates the inter-communicating compone…
Many science and industry IoT applications necessitate data processing across the edge-to-cloud continuum to meet performance, security, cost, and privacy requirements. However, diverse abstractions and infrastructures for managing resources and tasks across the edge-to-cloud scenario are required. We propose Pilot-Edge as a common abstraction for resource management across the edge-to-cloud continuum. Pilot-Edge is based on the pilot abstraction, which decouples resource and workload management, and provides a Function-as-a-Service (FaaS) interface for application-level tasks. The abstraction…
In this paper we describe our work on designing a web based, distributed data analysis system based on the popular MapReduce framework deployed on a small cloud; developed specifically for analyzing web server logs. The log analysis system consists of several cluster nodes, it splits the large log files on a distributed file system and quickly processes them using MapReduce programming model. The cluster is created using an open source cloud infrastructure, which allows us to easily expand the computational power by adding new nodes. This gives us the ability to automatically resize the cluste…
We present an Edge-as-a-Service (EaaS) platform for realising distributed cloud architectures and integrating the edge of the network in the computing ecosystem. The EaaS platform is underpinned by (i) a lightweight discovery protocol that identifies edge nodes and make them publicly accessible in a computing environment, and (ii) a scalable resource provisioning mechanism for offloading workloads from the cloud on to the edge for servicing multiple user requests. We validate the feasibility of EaaS on an online game use-case to highlight the improvement in the QoS of the application hosted on…
The Internet of Things (IoT) is offering unprecedented observational data that are used for managing Smart City utilities. Edge and Fog gateway devices are an integral part of IoT deployments to acquire real-time data and enact controls. Recently, Edge-computing is emerging as first-class paradigm to complement Cloud-centric analytics. But a key limitation is the lack of a platform-as-a-service for applications spanning Edge and Cloud. Here, we propose ECHO, an orchestration platform for dataflows across distributed resources. ECHO's hybrid dataflow composition can operate on diverse data mode…
Cloud computing is a new technology widely studied in recent years. Now there are many cloud platforms both in industry and in academic circle. How to understand and use these platforms is a big issue. A detailed comparison has been presented in this paper focused on the aspects such as the architecture, characteristics, application and so on. To know the differences between open source and close source in cloud environment we mention some examples for Software-as-a-Service, Platform-as-a-Service, and Infrastructure-as-a-Service. We made comparison between them. Before conclusion we demonstrat…
Graphical simulations are a cornerstone of modern media and films. But existing software packages are designed to run on HPC nodes, and perform poorly in the computing cloud. These simulations have complex data access patterns over complex data structures, and mutate data arbitrarily, and so are a poor fit for existing cloud computing systems. We describe a software architecture for running graphical simulations in the cloud that decouples control logic, computations and data exchanges. This allows a central controller to balance load by redistributing computations, and recover from failures. …
Through the 1990s to 2012 the internet changed the world of computing drastically. It started its journey with parallel computing after it advanced to distributed computing and further to grid computing. And in present scenario it creates a new world which is pronounced as a Cloud Computing [1]. These all three terms have different meanings. Cloud computing is based on backward computing schemes like cluster computing, distributed computing, grid computing and utility computing. The basic concept of cloud computing is virtualization. It provides virtual hardware and software resources to vario…
Cloud computing is a model for enabling on-demand network access to a shared pool of computing resources, that can be dynamically allocated and released with minimal effort. However, this task can be complex in highly dynamic environments with various resources to allocate for an increasing number of different users requirements. In this work, we propose a Cloud architecture based on a multi-agent system exhibiting a self-adaptive behavior to address the dynamic resource allocation. This self-adaptive system follows a MAPE-K approach to reason and act, according to QoS, Cloud service informati…
A non-invasive, cloud-agnostic approach is demonstrated for extending existing cloud platforms to include checkpoint-restart capability. Most cloud platforms currently rely on each application to provide its own fault tolerance. A uniform mechanism within the cloud itself serves two purposes: (a) direct support for long-running jobs, which would otherwise require a custom fault-tolerant mechanism for each application; and (b) the administrative capability to manage an over-subscribed cloud by temporarily swapping out jobs when higher priority jobs arrive. An advantage of this uniform approach …
Requirements and criteria for selection of cloud platform and platform visualization are stated by which optimal cloud products will be chosen for the Republic of Kazakhstan e-Government considering quality-price ratio, and also the framework of information and communication architecture will be introduced.
Distributed dataflow systems enable data-parallel processing of large datasets on clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. Yet, selecting appropriate cloud resources for dataflow jobs - that neither lead to bottlenecks nor to low resource utilization - is often challenging, even for expert users such as data engineers. We present C3O, a collaborative system for optimizing data processing cluster configurations in public clouds based on shared historical runtime data. The shared data is utilized for predicting the runti…
Graph analytics for large scale graphs has gained interest in recent years. Many graph algorithms have been designed for vertex-centric distributed graph processing frameworks to operate on large graphs with 100 M vertices and edges, using commodity clusters and Clouds. Subgraph-centric programming models have shown additional performance benefits than vertex-centric models. But direct mapping of vertex-centric and shared-memory algorithms to subgraph-centric frameworks are either not possible, or lead to inefficient algorithms. In this paper, we present three subgraph-centric distributed grap…
Distributed computing remains inaccessible to a large number of users, in spite of many open source platforms and extensive commercial offerings. While distributed computation frameworks have moved beyond a simple map-reduce model, many users are still left to struggle with complex cluster management and configuration tools, even for running simple embarrassingly parallel jobs. We argue that stateless functions represent a viable platform for these users, eliminating cluster management overhead, fulfilling the promise of elasticity. Furthermore, using our prototype implementation, PyWren, we s…
We show that distributed Infrastructure-as-a-Service (IaaS) compute clouds can be effectively used for the analysis of high energy physics data. We have designed a distributed cloud system that works with any application using large input data sets requiring a high throughput computing environment. The system uses IaaS-enabled science and commercial clusters in Canada and the United States. We describe the process in which a user prepares an analysis virtual machine (VM) and submits batch jobs to a central scheduler. The system boots the user-specific VM on one of the IaaS clouds, runs the job…
