The recent active development of Internet of Things (IoT) solutions in various domains has led to an increased demand for security, safety, and reliability of these systems. Security and data privacy are currently the most frequently discussed topics; however, other reliability aspects also need to be focused on to maintain the smooth and safe operation of IoT systems. Until now, there has been no systematic mapping study dedicated to the topic of interoperability and integration testing of IoT systems specifically; therefore, we present such an overview in this study. We analyze 803 papers from four major primary databases and perform detailed assessment and quality check to find 115 relevant papers. In addition, recently published testing techniques and approaches are analyzed and classified; the challenges and limitations in the field is also identified and discussed. Research trends related to publication time, active researchers, and publication media are presented in this study. The results suggest that studies mainly focus only on general testing methods, which can be applied to integration and interoperability testing of IoT systems; thus, there are research opportunities to develop additional testing methods focused specifically on IoT systems, so that they are more effective in the IoT context.
With the rapid growth of the contemporary Internet of Things (IoT) market, the established systems raise a number of concerns regarding the reliability and the potential presence of critical integration defects. In this paper, we present a PatrIoT framework that aims to provide flexible support to construct an effective IoT system testbed to implement automated interoperability and integration testing. The framework allows scaling from a pure physical testbed to a simulated environment using a number of predefined modules and elements to simulate an IoT device or part of the tested infrastruct…
The Internet of Things(IoT) will revolutionize the Future Internet through ubiquitous sensing. One of the challenges of having the hundreds of billions of devices that are estimated to be deployed would be rise of an enormous amount of data, along with the devices ability to manage. This paper presents an approach as a controller solution and designed specifically for autonomous management, connectivity and data interoperability in an IoT gateway. The approach supports distributed IoT nodes with both management and data interoperability with other cloud-based solutions. The concept further all…
Background: Testing IoT software is challenging due to large scale, volume of data and heterogeneity. Testing automation is a much-needed feature in the domain. Aims: The first goal of this research is to explore the requirements and challenges of IoT testing automation. The second goal is to integrate testing automation tools used in commercial software into the IoT context. Method: A systematic literature review is carried out to elicit requirements for testing automation in IoT. A design science approach is followed to build a testing automation tool for IoT applications written in the Node…
The Internet of things (IoT) is increasingly prevalent in domains such as emergency response, smart cities and autonomous vehicles. Simulation plays a key role in the testing of IoT systems, noting that field testing of a complete IoT product may be infeasible or prohibitively expensive. In this paper, we propose a domain-specific language (DSL) for generating edge-to-cloud simulators. An edge-to-cloud simulator executes the functionality of a large array of edge devices that communicate with cloud applications. Our DSL, named IoTECS, is the result of a collaborative project with an IoT analyt…
For Internet of Things (IoT) systems operating in areas with limited network connectivity, reliable and safe functionality must be ensured. This can be done using special test cases which are examining system behavior in cases of network outage and restoration. These test cases have to be optimal when approached from the testing effort viewpoint. When approached from the process viewpoint, in the sense that a business process supported by a tested system might be affected by a network outage and restoration, test cases can be automatically generated using a suitable model-based testing (MBT) t…
Quality and reliability metrics play an important role in the evaluation of the state of a system during the development and testing phases, and serve as tools to optimize the testing process or to define the exit or acceptance criteria of the system. This study provides a consolidated view on the available quality and reliability metrics applicable to Internet of Things (IoT) systems, as no comprehensive study has provided such a view specific to these systems. The quality and reliability metrics categorized and discussed in this paper are divided into three categories: metrics assessing the …
The continuous testing of small changes to systems has proven to be useful and is widely adopted in the development of software systems. For this, software is tested in environments that are as close as possible to the production environments. When testing IoT systems, this approach is met with unique challenges that stem from the typically large scale of the deployments, heterogeneity of nodes, challenging network characteristics, and tight integration with the environment among others. IoT test environments present a possible solution to these challenges by emulating the nodes, networks, and…
With the rapid growth in the number of IoT devices being added to the network, a major concern that arises is the security of these systems. As these devices are resource constrained, safety measures are difficult to implement on the edge. We propose a novel approach for the detection of IoT device attacks based on the use of formal modeling and mutation testing. Namely, we model the behavior of small IoT devices such as motion sensors and RFID reader as state machines with timeouts. We also model basic IoT attacks; namely, battery draining, sleep deprivation, data falsification, replay, and m…
