What Does Android Fragmentation Mean for Testers?
- Reading Time: 5 minutes
Though Apple users constitute a minority of mobile device users around the world, they’re a fairly predictable bunch when it comes to OS and firmware specifications. There are only about a handful of new Apple phones coming on the market in any given year, and users generally gravitate towards the most recent one.
Meanwhile, adoption of Apple OS updates is high. In June 2020, Apple estimated that more than 90% of all users were either using iOS 13 or iOS 12. From a telco operator’s perspective, this means that you’d expect remarkable consistency across the board in terms of user experience on your network.
With Android devices, things are a little trickier. In 2017, there were more than 24,000 unique Android devices on the market, and that number has been growing steadily since. Android boasts an 88% share of the global mobile device market, but its users are much less likely than Apple users to be utilizing the latest OS. In fact, less than 1% of Android devices in the world were running on the latest operating system in 2018, with plenty of users still running undated systems that are multiple years old.
For network operators, this presents a real problem: how do you provide consistent network quality and high QoS (quality of service) to users who are utilizing such a wide variety of hardware and software to access your network?
The answer begins in the test lab.
Android vs iOS
First things first, how did we get here? Why is fragmentation such a huge issue when it comes to Android devices in particular? That answer goes back about 13 years, when Google partnered with the Open Handset Alliance to develop Android as an open smartphone standard. The result is that most non-Apple smartphone manufacturers (from Samsung to Oppo to Huawei) have been making phones that run on Android for more than a decade. While the open standard has been advantageous for diversity and accessibility within this particular technological realm, it’s also created an environment that’s difficult to regulate. Manufacturers and carriers alike struggle to send out consistent updates for Android phones. From user to user, the Android landscape is filled with inconsistencies and security vulnerabilities. There’s plenty of content online detailing the issues that this state of affairs presents for mobile developers (essentially having to build multiple versions of an app in order to ensure interoperability), but telco testers are just as significantly impacted. Since many of your users will be connecting to your network via outdated and potentially buggy software and hardware, you have to make sure that each new network update you push out doesn’t interrupt service for users on outdated devices. Of course, this is on top of verifying conformance for a host of different devices that’s increasing in number every year. Given that most testers are already overloaded with use cases requiring verification, Android fragmentation only makes things worse.How Fragmentation Impacts Network Quality
So, as we’ve outlined it above, fragmentation can make high test quality/coverage, and thus high network quality, difficult in two ways:
- Huge numbers of unique devices in your network require high volumes of conformance testing.
- OS updates to devices aren’t performed evenly or consistently, adding complexity to conformance regression tests resulting from operator-side network updates.
For the first bullet point, it really is just a question of volume. Conformance and acceptance testing on tens of thousands of different individual Android devices simply isn’t feasible, which means you have to pick a number that you’re comfortable with. If we say that the goal is to cover 95% of your users, a European operator would have to test on 386 different phones. In North America, it’s only 324, and in Asia, it’s 534.
Let’s assume that the average telco tester can run through the relevant suite of tests for about 8 phones per day, and you decide that 300 unique phones is enough coverage. That means that the device acceptance tests for the initial rollout of a new service would require more than 37 working days.
At a moment when time-to-market is of paramount concern to most operators, and DevOps practices, such as monthly network releases, becomes a norm in telco industry, it means that network release is already outdated by the time device testing is completed. This is simply untenable. Add in regression tests, as well as more complex workflows for managing users with outdated operating systems, and you’re looking at a scenario where high test coverage—and thus high network quality—seems impossible.
Can Automation Help?
The short answer here is “yes.” Automated, non-intrusive tests (i.e. keyword-based tests that don’t require rooting or jailbreaking devices) give testers the power to cover a huge amount of device and OS diversity in record time. By using a testing solution that’s able to automate a critical mass of unique devices, you can run through hundreds of verification workflows in a matter of hours, rather than weeks. For Android devices in particular, your typical suite of tests might involve:- Call handling
- SMS handling
- Device status handling and verification
- MMI dialogs
- DTMF
- Speed tests
Categories
Tags
Thomas Groissenberger
Thomas has more than 25 years of experience in software, testing and management in Europe and the US. He is the Founder and CEO at SEGRON and he is a member of the Forbes Technology an Invitation-Only Community for World-Class CIOs, CTOs, and Technology Executives.
Recent Posts
Interested in our Products ?
