How To: 24/7 Regression Testing

how-to-24-7-regression-testing

Let’s talk about people for a second: by and large, they can’t and shouldn’t work 24 hours a day, seven days a week. They rightly prefer to stick to working hours, and when their jobs call for them to work outside of that time they’re usually provided extra compensation accordingly.

On top of that, one human being can really only do one thing at a time (studies show that people are actually really bad at multi-tasking), which means, for instance, they can either be running tests or fixing bugs at any given time—but not both.

Robots, on the other hand, don’t have these problems. They don’t ask for vacations or overtime, and they have no problem working on weekends. What’s more, they do the same things the same way over and over again without complaint or deviation.

Meaning that the results of their work are reliably standardized. Sure, they tend to lack the kinds of experience-driven and creative problem solving skills that human engineers bring to the table, but when it comes to rote and repetitive tasks they can’t be beat.

This makes them a perfect fit for something like continuous regression testing.

The Challenge

For most telco operations, the value of regression testing is pretty obvious. Sure, there will always be a mad dash to verify functionality and conformance in the immediate lead-up to a network update or a new product release

But if you’re able to continue running tests on your existing infrastructure as you go, you stand to find bugs and gaps in service that might not have been uncovered until the next big testing push. In this way, regression testing (in addition to uncovering bugs) makes future tests smoother and more efficient, which has the potential to speed up time-to-market.

The trouble here is that telco operators are necessarily working with a finite set of resources, and regression testing has a way of eating into those resources in a fairly significant way. There’s only so many hours in the week, and each hour that a given engineer spends on regression testing is an hour that they can’t use to actually fix uncovered bugs or focus on design decisions for new products.

To make matters more difficult, some systems can’t be tested during normal working hours for any number of business reasons, meaning that you’re potentially going to pay disgruntled testers serious overtime for ongoing service verification.

Suffice it to say that many businesses who otherwise see the value of ongoing regression tests aren’t willing or able to set aside the valuable engineer hours each week that this kind of workflow would imply.

The Solution

The situation we described above isn’t terribly uncommon—and, in fact, SEGRON has worked with major European telco operators who found themselves in this very position. So how did they create a testing environment in which 24/7 regression testing was feasible?

Simple: they used robots. This meant that, to begin with, they had to outline the use cases that would be best suited to regression testing—i.e. use cases that were simple and prevalent enough that they could be carried out quickly and repeatably on an ongoing basis (think VoLTE or VoWiFi verification).

Once those test cases were sketched out, they had to automate those tests using an appropriate test framework and equipment. In this case, they used SEGRON’s Automated Test Framework (ATF), plus 2 PCS operator SIM cards and two standard mobile phones (iOS or Android).

Crucially, these were out-of-the-box devices, rather than rooted or jailbreaked devices, in order to ensure that test conditions matched actual end user conditions as closely as possible.

Setting up the test environment took less than a day, and the tests could be easily configured to meet the specific needs of the operator—which in this case meant integrating the test framework with a centralized signal monitoring system as well as a series of audio analyses.

After setup was complete, the company was able to run ongoing tests both during and outside of normal business hours. This means that nights and weekends—when most human testers would be loath to come into the office—the robots kept chugging along.

Ultimately, these tests uncovered a number of bugs that would have been hard to catch in a normal service verification process—and those bugs were sent back to human engineers who were able to pinpoint their locations and work to address them.

In this way, the company was able to stretch its resources to make ongoing regression possible.

The Benefits of Continuous Regression

In the example above, SEGRON’s client was able to improve their standing a top telco operator in Europe with their increased ability to find and resolve bugs. Based on our experience, this is a fairly typical outcome for introducing automation into a regression testing environment.

We’ve alluded to this fact above, but, in general, automated 24/7 regression testing tends to have a few distinctive benefits for telco operators:

  • Increased opportunities to dedicate network engineer competencies and skills to performing root analyses and taking steps to fix bugs;
  • Major savings in terms of testing costs (due to the decreased need for ongoing human test resources);
  • Improved test coverage and test quality (due to the unvarying, repeatable nature of automated tests);
  • Shorter time-to-market.

If there’s an overarching theme to these benefits, it’s that automation for this use case gives you the flexibility to put your resources where they can do the most good. To the extent that you’re striving for higher test coverage, you’re working to automate in an area where automation can be especially impactful.

By the same token, by taking testing tasks off the plates of your engineers, you’re freeing up those resources to provide value elsewhere. It should be pretty easy to see how this would result in higher network quality overall, and thus better customer retention and an improved bottom line. It’s just a matter of letting the robots do their jobs.

Interested in our Products ?

Scroll Up

pencil Download your White Paper now

TECHNICAL PROJECT MANAGER

You will be a part of the SEGRON Project Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation.

As technical project manager, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases and you have to setup and align related project activities together with our customers.
You should have a strong knowledge and background in the telco environment paired with project management skills to align projects, track the related KPI’s further having a strong view also on the financial performance.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible
Team player:
• collaborates with others
• teach new skills to others and write tutorials
Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Several years of experience in Telecom Domain.
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• IMS/SDM/EPC domain knowledge
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack.
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge.

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus.
• Robot Framework / Python

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• PM Certification or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2500 EUR (depends on experience)

DEVELOPER TEST AUTOMATION

You will be a part of the SEGRON Test Automation Development Team. This will be an amazing opportunity for someone to be part of a team working in next generation cutting edge technologies such as 5G, IOT, DevOps, CI/CD and test automation using Python, Robot.

As test automation developer, you will be responsible for understanding the different customer requirements and latest technologies around 5G, VoLTE use cases, building automation tools, framework, creating reusable python libraries for test automation.
You should have a strong knowledge and hands-on experience in test automation framework (Robot) and Python scripting language. Further you must continuously renew the automation & review guidelines set within the team. Ensure developed reusable keywords & test scripts meet the automation guideline.

Soft Skills:
Problem Solving skills:
• Analytical & technical mindset
• Good in abstraction i.e. seeing the “big picture”
• Self-Motivated & Responsible 


Team player:
• collaborates with others
• teach new skills to others and write tutorials


Others:
• Good Communication Skills
• Flexible/Entrepreneurial

Hard Skills:
• Experience in developing reusable test automation scripts using Python and Robot Framework
• Several years of experience in Telecom Domain
• Good expertise on 5G/4G/3G technologies and should be familiar with the network call flows
• Test Case development
• Mobile development
• Design Patterns
• Web development/testing
• IMS/SDM/EPC domain knowledge
• Hand-on experience in Docker containers and Kubernetes.
• Sound understanding of Telco Cloud technologies such as VNFI, Open Stack
• Sound knowledge in Linux/CentOS, TCP/IP stack
• DevOps knowledge

Nice to have skills:
• Understanding of 5G network elements and interfaces is plus
• Wireshark packet analysis
• Experience in Protocol testing including HTTP2, SIP, MAP, Diameter

Specific additional information:
• You have a Telecommunications/Electronics/SW/Computer master’s degree or equivalent through experience
• Very high English proficient

Others:
• Full time job
• Variety of financial and non-financial benefits
• Offered salary: from 2000 EUR (depends on experience)