How (and Why) You Should Test on Flagship Devices

According to Apple, in the last quarter of 2019, the recently-released iPhone 11, iPhone 11 Pro, and iPhone 11 Pro Max made up nearly 70% of all American iPhone sales. It’s often been remarked that any data Apple happens to share should be taken with a grain of salt (they don’t release information about how many units they actually move), but even if this trend is exaggerated, it’s still fairly telling. In the US at least, mobile users are going out of their way to seek out the latest and greatest smartphones—and the odds are pretty high that some of those mobile users are subscribers on your network.

So, imagine their chagrin when they activate their new phone—excited to use its OLED screen and triple camera system—only to find themselves mired in network connectivity issues. iPhones are such high-profile devices that there would be an immediate outcry, but with new phones from other manufacturers, it might take a while for the issues to come to light. By that time, you’ve already lost the support of many of these users.  


How Out-of-the-Box Smartphones Impact Test Quality

The situation we outlined above might seem grim, but as luck would have it, there are proven ways to minimize that. Step one: test on the devices that users are utilizing to connect to your network. This sounds simple enough, but testers are often tempted to go the other way due to time pressure or perceived ease of automation. Usually, this takes the form of testers making use of devices that have been subject to rooting or jailbreaking. Considering the increasing complexity of the average telco network, it’s easy to imagine why someone would try to take a shortcut by mainly hacking devices to install testing software, but the result is tests that are less reliable than tests performed on devices out-of-the-box. Why? Because by making changes to the device iOS, you’re changing it just enough that it’s no longer a representative sample of other devices of its make, model, and firmware version. As a result, tiny differences in functionality can begin to add up to significant discrepancies.

Luckily, you don’t need to root or jailbreak your devices to automate testing. Instead, you can connect out-of-the-box devices—identical to the ones your subscribers are using—to a test server that primarily uses robotic process automation to dial numbers, perform calls, send SMS messages, and track the results in terms of both passage/failure of tests and protocol-level activity. These devices slot into automation flows just as seamlessly (we would often argue more seamlessly) then their rooted counterparts, but they don’t present the same downsides. Instead, they provide you with the closest possible recreation of real network usage conditions. 


The Importance of Flagship Devices

When it comes to testing on flagship devices, in particular, the rationale is essentially the same as the reasoning that was presented above. If your users are flocking to flagship devices as they’re released, they’re going to expect high network quality once they connect with their new devices. If you haven’t tested using these same devices, there’s a chance that something can go wrong, and that they won’t be able to place calls, use data, or otherwise use their phones as they expect to. Even when it comes to less high profile devices, that desire is still going to be there, and without proper service verification, as new devices enter the market, you’re going to run into issues. As technology changes and evolves, this will only become more true. To wit, the first generation of proper 5G-enabled phones has been trickling into the market for about a year now—and they’re equipped with different antennae and radio hardware than the prior generation of smartphones, such that they can connect to the new spectrums designated for millimeter-wave use. This presents a challenge for testers since there’s no realistic way to replicate these changes without using the devices themselves.

And this line of reasoning doesn’t just extend to phones. As we enter the IoT (internet of things) era, the number of cutting edge connected devices is increasing every year—putting considerable pressure on telco testers. Because IoT use cases can often be extremely high leverage (think of the eCall modem in a modern EU-based car that facilitates emergency calls in the case of an accident), the stakes are high, and it’s essential to have workflows in place, where you can test on the latest devices as efficiently as possible. Otherwise, your test lab runs the risk of missing important issues that could impact your subscribers. 

Automating Flagship Devices

As we said above, telco testers often look to rooting and jailbreaking to incorporate devices more easily into automation flows. After all, if you can’t test flagship devices within the context of test automation, there’s little hope of maintaining high test coverage or throughput. Thus, you might still be thinking of rooting as a last resort for testing on the absolute latest phones. But does rooting really speed up the process? Not necessarily. In SEGRON’s ATF (Automated Test Framework), it’s actually possible to incorporate new devices into the framework within a matter of hours. In this way, testers can incorporate the latest devices into their test lab operations without a hitch.

As you can probably guess from the preceding, once they’ve been automated in this way, there’s no need to root them. Instead, you can simply orchestrate tests, as usual, no doubt putting extra emphasis on conformance tests to ensure that any new firmware specifications play nicely with your network. Not only does this put you on a path to higher test coverage, but it also puts you in a position to address your subscribers’ needs proactively. With any luck, this can give you a competitive advantage over other operators in your area who might not be verifying functionality for the flagship iOS or Android devices that are just coming onto the market.


Download our latest eBook and Learn more about how Test Automation can speed up Time-to-Market, improve CX and more.

Device Testing
Jari Nurminen

Jari Nurminen

Jari has over 20 years of experience in software development, test automation and telecommunications. His expertise includes delivering Telecom and ICT solutions to major European carriers. As Chief Technology Officer (CTO), Jari leads the operation for the whole Automatic Test Framework ecosystem and the R&D department at Segron.

Interested in our Products ?

Scroll to Top