What Is the Future of IoT Testing?

Lots of articles have argued that the first ever internet of things (IoT) device was a soda machine at Carnegie Mellon that was connected to the early internet in the 1980s. The machine was famously automated to let users on the local network see how recently the machine had been filled (and thus how cold the soda was).

It’s a great story—but one device does not an internet make. In other words, what we in the modern era understand as the IoT isn’t really about individual devices; it’s about a tremendous volume of different devices all working together in tandem.

Testing conformance for individual devices will be critical for network operators, but it will only be one part of an equation that includes load testing for large networks of devices, conformance testing for different layers of applications and sensors, plus dealing with emergency use cases.

As a result, telco testers will soon find themselves in need of new strategies—and potentially new technologies—for ensuring high quality of service (QoS) for users.

Conformance Challenges

One of the most obvious challenges that presents itself in the current IoT landscape from a testing perspective is that the architecture of most IoT systems is a complex hash of heterogeneous technologies. IoT devices tend to work with a number of layers:

• Sensor
• Application
• Network
• Backend

And there’s no guarantee that those layers will be designed to play nicely with one another. For instance, you could easily have industrial IoT (IIoT) deployments in which the devices, applications, and middleware are all coming from different vendors, who might not be operating with the same technical specifications in mind.

In this case, you might find that each element in the IoT environment was working correctly, but the entire system wasn’t transferring production data back to the control tower effectively.

The implication here for testers is that, more so than in many other use cases, testing the entire process end-to-end is crucial. Without doing so, you risk the possibility that there are conformance failures between layers that will prevent the devices from networking together successfully in the field.

Once you’ve adopted an end-to-end mindset for IoT testing, you need to make sure that your test cases reflect it. So, on top of the standard set of use cases for the devices (validating network connectivity, verifying data packets/measuring packet loss, and validating synchronization, transmission frequency, multiple-request handling, etc.).

You’ll need to recreate the entire set of systems and subsystems to make sure that the whole end-to-end flow is actually functioning within the confines of your network. Needless to say, this can become extremely resources-intensive, especially if you’re trying to test all of the use cases by hand.

New Network Expectations

Sure, one of the biggest challenges that comes with IoT testing is the relative complexity, but that complexity is just one piece of the puzzle. An equally important piece of that puzzle is the increase in load that networks running IoT devices are likely to face.

Though most IoT devices will operate on low power and low throughput, there will be tons of them trying to connect to your network in order to constantly transmit information. As of about a year ago, estimates put the number of IoT devices in operation around the globe around 7 billion.

And that number is only going to keep rising as this nascent technology grows. If the hype around 5G is to be believed, we could see exponential growth in this number within a few short years.

To truly ensure that their networks are ready for full IoT deployments, testers will have to find a way to replicate this peculiar type and volume of network traffic within their test environments.

This will begin with larger scale deployment of test automation (such that the relevant devices can be controlled through an agile interface), and will likely involve the creation of complex testing environments that differ radically from what telco testers are used to.

Devices will have to be able to connect to your network successfully and continue transmitting information without getting booted off accidentally, and you’ll have to ensure that your network is allocating bandwidth appropriately under these new expectations and constraints.

Going Beyond End-to-end

Because of the high degree of complexity involved in testing IoT deployments, testers are also in somewhat of a tricky position when it comes to uncovering the root causes of any bugs that they encounter. For this reason, it’s also potentially worth moving beyond end-to-end workflows in your testing.

This could mean utilizing signalling analysis and CDR data from various network elements to ensure that there are no tuning issues, for instance. By moving beyond the level of success/failure and into what’s actually happening at the signalling level, you can gain a more comprehensive view of how your network is interacting with these devices.

The result of this more comprehensive view? The ability to detect potential QoS risks before they’ve actually caused a device to disconnect or a packet to get lost.

As the IoT becomes more entrenched in the technology and telco landscapes, with think that the future of testing for this area will revolve around attempts to solve the challenges we’ve been laying out above.

The trick, of course, will be not just to solve them—but to solve them efficiently at scale. To that end, testers are going to have to think long and hard about what kind of testing technology they want to employ going forward, as well as how they want to structure their test operations.

Are there ways that complex test cases can be streamlined and sped up to improve throughput? Can regression tests be made less time intensive and therefore more feasible for IoT use cases? Only time and test labs will tell.