Are Tests Run on Rooted Devices Reliable?

Sometimes, the OS that comes built into an Apple or Android device can seem like it’s actively preventing you from doing what you want. And, to some extent, that’s true—Apple only lets you download the apps that are in the App Store because there are some actions that they want to prevent you from taking.

As an end user, this often seems capricious and arbitrary. As a tester, it feels similar: why shouldn’t you be allowed to make the necessary changes and additions to the phone’s functionality to make end-to-end testing easier?

Because some automation solutions effectively require that you make use of mobile devices that have been modified to allow changes to the OS, i.e. devices that have been rooted or jailbreaked. The question, then, is whether tests run under these circumstances can still provide testers with the information they need to verify service.

What Is Rooting/Jailbreaking?

Okay, let’s back up a second so that we can define our terms. Though rooting and jailbreaking are, in fact, two distinct processes, they both involve removing hardware restrictions from Android or iOS devices.

When you’re removing those restrictions on an iPhone, it’s referred to as jailbreaking, which a reference to the use of “chroot jails” to isolate processes in Unix. The end result is that you have elevated permissions and can suddenly run applications that wouldn’t have been available to you otherwise.

Most functionality (like calling and running built-in apps) is maintained, but some security featured won’t run anymore and the OS won’t be able to update consistently.

Rooting, which is the equivalent process on Android, actually takes you one step further: it doesn’t just elevate your permissions—it gives you root access to the Android sub-system.

From there, you can actually modify the OS arbitrarily (which isn’t possible on a jailbreaked iOS device), which ostensibly elevates the risk that you’ll “brick” the device (i.e. make it unusable) with any change that you make.

Why Jailbreak?

For software testers, rooting and jailbreaking have historically presented added freedom to create a test environment that suits their needs. LinkedIn’s test-butler, for example, which is designed to stabilize emulated Android devices for a cleaner test environment, can actually be run on rooted Android devices.

This means that testers can ostensibly take better control over their test environments (without worrying about crash and ANR pop-ups, WiFi radio going to sleep, animations, etc.), without dealing with any of the downsides of running tests on emulated devices.

When it comes to the telco domain, the rationale is usually similar: testers want to automate their conformance tests, say, and the solutions they’re considering offer up rooted devices as a way to do this efficiently and cleanly at scale.

Unfortunately, this tactic introduces a whole host of risks into the testing process. Let’s say you’re attempting to verify that native Android applications run correctly on your network: you have a suite of test cases in your library that run through the functionality of the app step by step, from logging in to performing some application-specific tasks and logging back out.

Because the OS on the rooted device doesn’t actually correspond to the OS being used by end users, the results of your test will be inconclusive. If a particular test fails, you’ll be at a loss to ascertain whether the root cause is a problem in your network, the device out-of-the-box, or the way the rooted device has been altered.

Because rooted devices don’t receive the periodic updates that get sent out to devices that haven’t been altered in this way, tests run on rooted devices are essentially tests on obsolete technology.

This is a tradeoff that many testers wind up taking (since the odds of any individual software update having a huge impact on, say, network connectivity aren’t that high), but it certainly does add risk for testers.

After all, subtle changes in the device can begin to have an outsized impact as they accumulate—especially as standards for latency get more and more stringent.

Rooted vs. Out-of-the-Box

Okay, we’ve gone over some of the reasons why you might want to test on a rooted device, as well as some of the risks that that might entail. From our perspective, these risks make testing on rooted devices a less-than-attractive option.

Out-of-the-box iOS and Android devices, by contrast, help you to virtually eliminate the risks that we described above by ensuring that your test environment resembles real network usage conditions as closely as possible.

Since they’ll receive regular software updates like a subscriber’s phone would, you won’t have to worry about testing on outdated phones. Likewise, because you haven’t changed the OS at all, when you find a bug you can uncover the root cause much more easily, since you know that it’s either an issue with the device itself or your network.

But if you’re under pressure to automate your tests, you might be wondering whether out-of-the-box devices actually present a viable alternative. After all, isn’t the whole point of rooting or jailbreaking a device to empower testers to run code that makes it easier to automate application tests, acceptance tests, conformance tests, etc.?

Even if they’re not 100% reliable, couldn’t you utilize them as part of your broader testing flow? Well, sure—but these days it’s possible to control out-of-the-box devices with an external interface in order to run through test cases automatically.

In this way, you can automate test cases without having to gain root access to the devices’ operating systems, meaning that you can automate without incurring any of the risks we discussed above.

This type of solution will scale just as easily as testing with rooted devices (if not even more easily), since you can automatically run through test cases without manual intervention—potentially resulting in more accurate, and thus more value-additive, test automation.

At the end of the day, the best way to make sure you’re testing the real thing—i.e. a device that’s 100% representative of the technology that your subscribers are actually using—is to avoid hacking your devices to install an invasive testing app.