Is It Time for Telco Testing to Go Beyond End-to-End?

is-it-time-for-testing-in-telecom-operations-to-go-beyond-end-to-end

For many telco operators, testing can seem like an onerous requirement. It’s often costly and time consuming, and as telecom networks grow more complex and customer use cases and devices become increasingly fragmented, verifying service with any level of confidence is harder than ever.

Because of this high degree of complexity, testers need to achieve higher test coverage than ever before in order to maintain network quality—resulting in the relatively widespread adoption of end-to-end testing among those in the industry. Rather than testing voice protocols and Wi-Fi connectivity from a handful of user devices, testers are walking through entire systems and subsystems in the ways that users are likely to do.

This has the potential to improve testing quality and coverage, resulting in improved quality of service, but it also presents some questions: in an era of increasing use case complexity, how can end-to-end testing be performed in an efficient and scalable manner?

Is end-to-end testing enough to meet service verification needs in the modern era, or are there additional steps that businesses could be taking to better address latent service issues?

The Case for End-to-End Testing

Let’s back up a step: end-to-end testing is a fairly common buzzword, but what does it actually mean? In software development, end-to-end refers to the verification not just of functionality within the app or program itself, but with regard to the apps, programs, and interfaces that it ultimately interacts with.

From a telecom perspective, this makes an intuitive kind of sense: an end user on his or her smartphone will variously access 4G, for instance, not in an abstract way, but using particular applications. Perceived service quality will depend on how well those apps function.

Even if your 4G service, or your VoIP, or mobile broadband, or anything else that you might be offering as a service provider is ostensibly “strong,” a bug or malfunction that occurs somewhere between the cell tower and Venmo can still convince a user that your network is low quality.

So, in one sense, end-to-end is a no-brainer for telcos. On the other hand, it does necessarily increase the scope of your functional testing. Your typical test engineer can only get through half a dozen use cases per day on average, which means that a test framework designed around developing a large number of use cases for verification is going to be so time and labor intensive as to seem utterly overwhelming.

This tends to lead testers away from true end-to-end testing and into more efficient forms of service verification that can be scaled up more easily. Sure, it’s possible to automate an end-to-end framework in such a way as to maintain integrity while improving your test coverage

But in many areas it’s more common to see the use of simulated tests or tests on rooted devices standing in for real end-user devices, resulting in disconnect between lab conditions and actual network usage.

Beyond End-to-End

If end-to-end testing stretches from the first moment of network usage for a given use case (e.g. logging into one’s email app while roaming on a 3G network) to the last (sending the email and getting confirmation of the send), it’s worth asking what, if anything, remains to be tested.

If the potential limitation of end-to-end testing that we sketched out above (i.e. that it’s too labor intensive) is the only issue that testers are likely to come across, then it seems like end-to-end per se—as opposed to the difficulty of truly implementing end-to-end frameworks—isn’t the problem. Thats certainly true to some extent; at the same time, however, it is possible to improve test coverage beyond end-to-end and gain value in the process.

Let’s look at a real-life example: several years ago, Google experimented with ways to reduce latency times for its users. It began with an end-to-end approach, redirecting users to whichever server was offering the lowest latency times at any given moment.

Strangely enough, they found that this approach didn’t yield the lowest possible latencies—in part because the queueing process for server allocation was causing slowdowns and in part because inflated latency times turned out to correlate more closely with particular interactions between the nodes than the conditions at the end points in particular.

Here, we see an example of what going beyond end-to-end could look like. Sure, this example isn’t squarely within the telecom domain, but it’s easy enough to imagine an equivalent case for a telco operator: in verifying voice functionality, for instance, you might look beyond the end-points (i.e. the two users having a phone call) and analyze signaling patterns, etc.

Many systems will continue to function even when one or more elements fail or run inefficiently, and this type of testing enables you to catch the kinds of failures or bug that aren’t immediately impacting functionality (but carry some latent risk).

Measuring ROI

At this point you might be thinking: “That’s all well and good, but test budgets are stretched thin as they are, and going beyond end-to-end hardly seems practical under current market conditions. Verifying service with current test coverage levels really has to cut it for now if I want a positive ROI.”

This is a reasonable objection. But let’s take it one step further: you can’t really stick with manual testing at all if you want a positive ROI, because reaching a critical mass of test coverage is becoming increasingly time consuming. If you want end-to-end testing, you need to automate your testing framework.

By automating your tests (using out-of-the-box devices), you can increase the number of uses cases verified each day from dozens to hundreds, resulting in better test coverage and reduced costs in the form of reduced person-hours. In this way, moving beyond end-to-end starts to become feasible: since tests can be performed quickly, you can essentially assume that your ROI goes up as your coverage improves (since better coverage should continue to improve your quality-of-service).

Thus, you put yourself in a position in which it’s feasible to analyze not just end-point results but signaling, alarms, logs, etc. in order to gain an even better command of what’s happening on your network.

Not only does this improve your odds of uncovering any latent bugs, it puts you in a position to create granular documentation that will cover not just test cases by mission critical data about the health of your services. In this way, testing beyond end-to-end becomes a strategic investment in short- and long-term quality-of-service improvements.

Interested in our Products ?

Scroll Up

pencil Download your White Paper now

TECHNICAL PROJECT MANAGER

You will be a part of the SEGRON Project Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation.

As technical project manager, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases and you have to setup and align related project activities together with our customers.
You should have a strong knowledge and background in the telco environment paired with project management skills to align projects, track the related KPI’s further having a strong view also on the financial performance.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible
Team player:
• collaborates with others
• teach new skills to others and write tutorials
Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Several years of experience in Telecom Domain.
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• IMS/SDM/EPC domain knowledge
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack.
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge.

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus.
• Robot Framework / Python

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• PM Certification or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2500 EUR (depends on experience)

DEVELOPER TEST AUTOMATION

You will be a part of the SEGRON Test Automation Development Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation using Python, Robot.

As test automation developer, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases, building automation tools, framework, creating reusable python libraries for test automation.
You should have a strong knowledge and hands-on experience in test automation framework (Robot) and Python scripting language. Further you must continuously renew the automation & review guidelines set within the team. Ensure developed reusable keywords & test scripts meet the automation guideline.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible 


Team player:
• collaborates with others
• teach new skills to others and write tutorials


Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Experience in developing reusable test automation scripts using Python and Robot Framework
• Several years of experience in Telecom Domain
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• Test Case development
• Mobile development
• Design Patterns
• Web development/testing
• IMS/SDM/EPC domain knowledge
• Hand-on experience in Docker containers and Kubernetes.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2000 EUR (depends on experience)