If you’re a smartphone manufacturer, you’ve likely encountered this issue: you’ve been working on a significant upgrade to your operating system, and you finally hit the ‘go’ button and push it out to your users. Most of your users get a notification asking them to update, they do so, and everything is fine. But some of your users try to install the update only to find that it crashes their system or has other unwanted consequences for their ability to use their phones. Some users, fearing precisely that scenario, likely won’t download the updates at all—potentially leaving themselves open to security vulnerabilities and quality of service (QoS) issues.
This situation certainly isn’t uncommon, but it’s a problematic state of affairs for telco businesses. As more test engineers in the telco space turn to tactics like CICD (continuous integration/continuous delivery—a testing tactic in which small updates are being pushed, built, tested, and delivered on a daily basis). However, the cycle of significant updates with patchwork adoption begins to change. Thus, instead of a mishmash of slightly different OS specifications across your users—resulting in security, compliance, and QoS issues—most users get small, periodic updates that don’t break anything.
This is just one example of how a particular testing tactic can have a direct, immediate impact on user experience. Though your test flows might feel a little abstracted from users’ realities on our devices or networks, it’s becoming clear that what you do on the test bench can reach your users in the form of something other than fluctuating jitter latency and packet loss.
As a baseline, you can start by automating tests as generally as possible using out-of-the-box smartphones, i.e., phones that haven’t been rooted or jailbroken. This puts you in a position to replicate the conditions under which your users will actually be operating your phones or accessing your network, such that you can be sure that you’re seeing exactly what they are.
What Impacts on Telco Networks?
Like we said above, most testers think of things like jitter and packet loss that can be directly measured and tested for as the most apparent factors impacting user experience. And, to some extent, that’s true: without a baseline level of network quality that allows users to make calls and use data without frustration, there’s little point in talking about the rest of it. At the same time, however, user experience goes beyond just those metrics. This is why some businesses have started tracking the quality of experience (QoE) in addition to just QoS (Quality of Service). This makes it possible to examine more intangible factors that might make or break the user experience.
When it comes to QoE, in addition to things like latency times and outages, users care a lot about how long it takes for issues (whether that’s their particular issues—the ones that lead them to call your customer service hotline—or systemwide problems) to get resolved. It might also include ease of use for anything that interacts with your backend OSS/BSS systems, like changing account settings or paying bills. This, again, could put testers in somewhat new territory, pushing their focus towards provisioning, BSS integrations, and specific UX questions in a way that wasn’t common even a few years ago. Integrating automated testing of web-based user portals and provisioning systems gives a clearer picture of how long actions take.
The Power of
From there, you can begin to get more sophisticated: with a testing framework that can automatically run tests on your OSS/BSS systems, you can start to create alignment between those systems and your QoE goals. For instance, by including things like invoicing and billing in your standard suite of regression tests. If you have a system capable of going “beyond end-to-end,” you can also start to test things like audio quality and speech path connections. For an area like this, there are, among others, three prominent use cases:
- Tone verification
- Announcements identification
- Speech path testing
Testing here might involve some extra elements (like an audio matrix and an isolator). It could include injecting audio samples into one side of the connection and recording the audio output on the other side of the connection. In that way, you’re able to expend resources on optimizing things that your customers will experience in a fairly direct way.
For audio QoE measurements, the global standard Perceptual Objective Listening Quality Analysis (POLQA) is used to model the speech path audio quality and produce the QoE as a mean opinion score (MOS). Combining such QoE estimates with technical QoS measures gives a clear picture of how network performance translates into customer satisfaction.
How to add QoE-focused tests without hurting
Luckily, in an automated testing environment, testers don’t have to worry about fitting in all of these test cases that might typically have fallen by the wayside. Instead, they can plug in a few keywords, sit back, and let the magic happen (until it’s time for aftercare and root cause analysis). The trick is to adopt the right testing tools. It might include:
- A test automation framework
- An audio matrix and isolators (for audio QoE testing)
- Shielded boxes and attenuators (for testing mobility-related use cases)
- Tracing tools like TCPdump and Wireshark (for going beyond end-to-end to capture signalling traces)
- Testcase databases and libraries
To employ some CICD workflow, you’ll also need a development repository and version control system like Git and an integration server like Jenkins (an open-source CI server that automatically builds and tests developer code after each commit). With the right tools set up, you can begin to create test flows that aren’t just about checking off boxes and verifying latency speeds—but are instead built around your customers and their needs. At the end of the day, it’s the customer’s priorities that determine your QoS, QoE, and ROI—a fact that your test flows need to reflect.