How Telco Test Automation Impacts User Experience

Hands on Tablet

If you're a smartphone manufacturer, you've likely encountered this issue: you've been working on a significant upgrade to your operating system, and you finally hit the ‘go' button and push it out to your users. Most of your users get a notification asking them to update, they do so, and everything is fine. But some of your users try to install the update only to find that it crashes their system or has other unwanted consequences for their ability to use their phones. Some users, fearing precisely that scenario, likely won't download the updates at all—potentially leaving themselves open to security vulnerabilities and quality of service (QoS) issues. 

 This situation certainly isn't uncommon, but it's a problematic state of affairs for telco businesses. As more test engineers in the telco space turn to tactics like CICD (continuous integration/continuous delivery—a testing tactic in which small updates are being pushed, built, tested, and delivered on a daily basis). However, the cycle of significant updates with patchwork adoption begins to change. Thus, instead of a mishmash of slightly different OS specifications across your users—resulting in security, compliance, and QoS issues—most users get small, periodic updates that don't break anything. 

This is just one example of how a particular testing tactic can have a direct, immediate impact on user experience. Though your test flows might feel a little abstracted from users' realities on our devices or networks, it's becoming clear that what you do on the test bench can reach your users in the form of something other than fluctuating jitter latency and packet loss.  

As a baseline, you can start by automating tests as generally as possible using out-of-the-box smartphones, i.e., phones that haven't been rooted or jailbroken. This puts you in a position to replicate the conditions under which your users will actually be operating your phones or accessing your network, such that you can be sure that you're seeing exactly what they are. 

What Impacts User Experience on Telco Networks?

Like we said above, most testers think of things like jitter and packet loss that can be directly measured and tested for as the most apparent factors impacting user experience. And, to some extent, that's true: without a baseline level of network quality that allows users to make calls and use data without frustration, there's little point in talking about the rest of it. At the same time, however, user experience goes beyond just those metrics. This is why some businesses have started tracking the quality of experience (QoE) in addition to just QoS (Quality of Service). This makes it possible to examine more intangible factors that might make or break the user experience. 

When it comes to QoE, in addition to things like latency times and outages, users care a lot about how long it takes for issues (whether that's their particular issues—the ones that lead them to call your customer service hotline—or systemwide problems) to get resolved. It might also include ease of use for anything that interacts with your backend OSS/BSS systems, like changing account settings or paying bills. This, again, could put testers in somewhat new territory, pushing their focus towards provisioning, BSS integrations, and specific UX questions in a way that wasn't common even a few years ago. Integrating automated testing of web-based user portals and provisioning systems gives a clearer picture of how long actions take. 

The Power of Test Automation

From there, you can begin to get more sophisticated: with a testing framework that can automatically run tests on your OSS/BSS systems, you can start to create alignment between those systems and your QoE goals. For instance, by including things like invoicing and billing in your standard suite of regression tests. If you have a system capable of going “beyond end-to-end,” you can also start to test things like audio quality and speech path connections. For an area like this, there are, among others, three prominent use cases: 

  • Tone verification 
  • Announcements identification 
  • Speech path testing 

Testing here might involve some extra elements (like an audio matrix and an isolator). It could include injecting audio samples into one side of the connection and recording the audio output on the other side of the connection. In that way, you're able to expend resources on optimizing things that your customers will experience in a fairly direct way. 

For audio QoE measurements, the global standard Perceptual Objective Listening Quality Analysis (POLQA) is used to model the speech path audio quality and produce the QoE as a mean opinion score (MOS). Combining such QoE estimates with technical QoS measures gives a clear picture of how network performance translates into customer satisfaction. 

How to add QoE-focused tests without hurting time-to-market?

Luckily, in an automated testing environment, testers don't have to worry about fitting in all of these test cases that might typically have fallen by the wayside. Instead, they can plug in a few keywords, sit back, and let the magic happen (until it's time for aftercare and root cause analysis). The trick is to adopt the right testing tools. It might include: 

  • A test automation framework 
  • An audio matrix and isolators (for audio QoE testing)  
  • Shielded boxes and attenuators (for testing mobility-related use cases)
  • Tracing tools like TCPdump and Wireshark (for going beyond end-to-end to capture signalling traces) 
  • Testcase databases and libraries 

To employ some CICD workflow, you'll also need a development repository and version control system like Git and an integration server like Jenkins (an open-source CI server that automatically builds and tests developer code after each commit). With the right tools set up, you can begin to create test flows that aren't just about checking off boxes and verifying latency speeds—but are instead built around your customers and their needs. At the end of the day, it's the customer's priorities that determine your QoS, QoE, and ROI—a fact that your test flows need to reflect. 

Interested in our Products ?







pencil Download your White Paper now

TECHNICAL PROJECT MANAGER

You will be a part of the SEGRON Project Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation.

As technical project manager, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases and you have to setup and align related project activities together with our customers.
You should have a strong knowledge and background in the telco environment paired with project management skills to align projects, track the related KPI’s further having a strong view also on the financial performance.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible
Team player:
• collaborates with others
• teach new skills to others and write tutorials
Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Several years of experience in Telecom Domain.
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• IMS/SDM/EPC domain knowledge
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack.
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge.

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus.
• Robot Framework / Python

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• PM Certification or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2500 EUR (depends on experience)

DEVELOPER TEST AUTOMATION

You will be a part of the SEGRON Test Automation Development Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation using Python, Robot.

As test automation developer, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases, building automation tools, framework, creating reusable python libraries for test automation.
You should have a strong knowledge and hands-on experience in test automation framework (Robot) and Python scripting language. Further you must continuously renew the automation & review guidelines set within the team. Ensure developed reusable keywords & test scripts meet the automation guideline.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible 


Team player:
• collaborates with others
• teach new skills to others and write tutorials


Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Experience in developing reusable test automation scripts using Python and Robot Framework
• Several years of experience in Telecom Domain
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• Test Case development
• Mobile development
• Design Patterns
• Web development/testing
• IMS/SDM/EPC domain knowledge
• Hand-on experience in Docker containers and Kubernetes.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2000 EUR (depends on experience)