What Does Android Fragmentation Mean for Testers?

Android Frag

Though Apple users make up a minority of mobile device users around the world, they’re a fairly predictable bunch when it comes to OS and firmware specifications. There are only about a handful of new Apple phones coming on the market in any given year, and users generally gravitate towards the most recent one.

Meanwhile adoption of Apple OS updates is high. In June 2020, Apple estimated that more than 90% of all users were either using iOS 13 or iOS 12. From a telco operator’s perspective, this means that you’d expect remarkable consistency across the board in terms of user experience on your network.

With Android devices, things are a little bit trickier. In 2017, there were more than 24,000 unique Android devices on the market, and that number has been growing steadily since. Android boasts an 88% share of the global mobile device market, but its users are much less likely than Apple users to be utilizing the latest OS. In fact, less than 1% of Android devices in the world were running on the latest operating system in 2018, with plenty of users still running un-updated systems that are multiple years old. 

For network operators, this presents a real problem: how do you provide consistent network quality and high QoS (quality of service) to users who are utilizing such a wide variety of hardware and software to access your network? 

The answer begins in the test lab.

Android vs iOS

First things first, how did we get here? Why is fragmentation such a huge issue when it comes to Android devices in particular?  That answer goes back about 13 years, when Google partnered with the Open Handset Alliance to develop Android as an open smartphone standard. The result is that most non-Apple smartphone manufacturers (from Samsung to Oppo to Huawei) have been making phones that run on Android for more than a decade.  While the open standard has been advantageous for diversity and accessibility within this particular technological realm, it’s also created an environment that’s difficult to regulate. Manufacturers and carriers alike struggle to send out consistent updates for Android phones. From user to user, the Android landscape is filled with inconsistencies and security vulnerabilities. There’s plenty of content online detailing the issues that this state of affairs presents for mobile developers (essentially having to build multiple versions of an app in order to ensure interoperability), but telco testers are just as significantly impacted. Since many of your users will be connecting to your network via outdated and potentially buggy software and hardware, you have to make sure that each new network update you push out doesn’t interrupt service for users on outdated devices.  Of course, this is on top of verifying conformance for a host of different devices that’s increasing in number every year. Given that most testers are already overloaded with use cases requiring verification, Android fragmentation only makes things worse.

How Fragmentation Impacts Network Quality

So, as we’ve outlined it above, fragmentation can make high test quality/coverage, and thus high network quality, difficult in two ways:

  • Huge numbers of unique devices in your network require high volumes of conformance testing.
  • OS updates to devices aren’t performed evenly or consistently, adding complexity to conformance regression tests resulting from operator-side network updates.

For the first bullet point, it really is just a question of volume. Conformance and acceptance testing on tens of thousands of different individual Android devices simply isn’t feasible, which means you have to pick a number that you’re comfortable with. If we say that the goal is to cover 95% of your users, a European operator would have to test on 386 different phones. In North America, it’s only 324, and in Asia, it’s 534.

Let’s assume that the average telco tester can run through the relevant suite of tests for about 8 phones per day, and you decide that 300 unique phones is enough coverage. That means that the device acceptance tests for the initial rollout of a new service would require more than 37 working days. 

At a moment when time-to-market is of paramount concern to most operators, and DevOps practices, such as monthly network releases, becomes a norm in telco industry, it means that network release is already outdated by the time device testing is completed. This is simply untenable. Add in regression tests, as well as more complex workflows for managing users with outdated operating systems, and you’re looking at a scenario where high test coverage—and thus high network quality—seems impossible.

Can Automation Help?

The short answer here is “yes.” Automated, non-intrusive tests (i.e. keyword-based tests that don't require rooting or jailbreaking devices) give testers the power to cover a huge amount of device and OS diversity in record time.  By using a testing solution that’s able to automate a critical mass of unique devices, you can run through hundreds of verification workflows in a matter of hours, rather than weeks. For Android devices in particular, your typical suite of tests might involve:
  • Call handling
  • SMS handling
  • Device status handling and verification
  • MMI dialogs
  • DTMF
  • Speed tests
By establishing your set of test devices with the appropriate keywords, suddenly you can automatically run through each step of the verification process without spending an exorbitant number of engineering hours. The result is that you cover the needs of more users without slowing down time-to-market.  So, what does Android fragmentation mean for testers? The simple answer is that it means manual device testing is a less feasible strategy now more than ever. As telco networks become more and more complex, the number of use cases that can be verified by hand in one day has been steadily dropping—and Android fragmentation only makes it worse.  However, by automating your testing, you can make Android fragmentation a non-issue. Not only that, but you can improve network quality for the vast majority of your users no matter what particular phone they happen to be using. If you can do that, you’ll earn their continued trust, and ultimately retain them as subscribers for a long time to come.

Interested in our Products ?







pencil Download your White Paper now

TECHNICAL PROJECT MANAGER

You will be a part of the SEGRON Project Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation.

As technical project manager, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases and you have to setup and align related project activities together with our customers.
You should have a strong knowledge and background in the telco environment paired with project management skills to align projects, track the related KPI’s further having a strong view also on the financial performance.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible
Team player:
• collaborates with others
• teach new skills to others and write tutorials
Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Several years of experience in Telecom Domain.
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• IMS/SDM/EPC domain knowledge
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack.
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge.

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus.
• Robot Framework / Python

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• PM Certification or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2500 EUR (depends on experience)

DEVELOPER TEST AUTOMATION

You will be a part of the SEGRON Test Automation Development Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation using Python, Robot.

As test automation developer, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases, building automation tools, framework, creating reusable python libraries for test automation.
You should have a strong knowledge and hands-on experience in test automation framework (Robot) and Python scripting language. Further you must continuously renew the automation & review guidelines set within the team. Ensure developed reusable keywords & test scripts meet the automation guideline.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible 


Team player:
• collaborates with others
• teach new skills to others and write tutorials


Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Experience in developing reusable test automation scripts using Python and Robot Framework
• Several years of experience in Telecom Domain
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• Test Case development
• Mobile development
• Design Patterns
• Web development/testing
• IMS/SDM/EPC domain knowledge
• Hand-on experience in Docker containers and Kubernetes.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2000 EUR (depends on experience)