Let’s talk about people for a second: by and large, they can’t and shouldn’t work 24 hours a day, seven days a week. They rightly prefer to stick to working hours, and when their jobs call for them to work outside of that time they’re usually provided extra compensation accordingly.
On top of that, one human being can really only do one thing at a time (studies show that people are actually really bad at multi-tasking), which means, for instance, they can either be running tests or fixing bugs at any given time—but not both.
Robots, on the other hand, don’t have these problems. They don’t ask for vacations or overtime, and they have no problem working on weekends. What’s more, they do the same things the same way over and over again without complaint or deviation.Meaning that the results of their work are reliably standardized. Sure, they tend to lack the kinds of experience-driven and creative problem solving skills that human engineers bring to the table, but when it comes to rote and repetitive tasks they can’t be beat.
This makes them a perfect fit for something like continuous regression testing.
The Challenge
For most telco operations, the value of regression testing is pretty obvious. Sure, there will always be a mad dash to verify functionality and conformance in the immediate lead-up to a network update or a new product release
But if you’re able to continue running tests on your existing infrastructure as you go, you stand to find bugs and gaps in service that might not have been uncovered until the next big testing push. In this way, regression testing (in addition to uncovering bugs) makes future tests smoother and more efficient, which has the potential to speed up time-to-market.
The trouble here is that telco operators are necessarily working with a finite set of resources, and regression testing has a way of eating into those resources in a fairly significant way. There’s only so many hours in the week, and each hour that a given engineer spends on regression testing is an hour that they can’t use to actually fix uncovered bugs or focus on design decisions for new products.
To make matters more difficult, some systems can’t be tested during normal working hours for any number of business reasons, meaning that you’re potentially going to pay disgruntled testers serious overtime for ongoing service verification.
Suffice it to say that many businesses who otherwise see the value of ongoing regression tests aren’t willing or able to set aside the valuable engineer hours each week that this kind of workflow would imply.
The Solution
The situation we described above isn’t terribly uncommon—and, in fact, SEGRON has worked with major European telco operators who found themselves in this very position. So how did they create a testing environment in which continuous regression testing was feasible?
Simple: they used robots. This meant that, to begin with, they had to outline the use cases that would be best suited to regression testing—i.e. use cases that were simple and prevalent enough that they could be carried out quickly and repeatably on an ongoing basis (think VoLTE or VoWiFi verification).
Once those test cases were sketched out, they had to automate those tests using an appropriate test framework and equipment. In this case, they used SEGRON’s Automated Test Framework (ATF), plus 2 PCS operator SIM cards and two standard mobile phones (iOS or Android).
Crucially, these were out-of-the-box devices, rather than rooted or jailbreaked devices, in order to ensure that test conditions matched actual end user conditions as closely as possible.
Setting up the test environment took less than a day, and the tests could be easily configured to meet the specific needs of the operator—which in this case meant integrating the test framework with a centralized signal monitoring system as well as a series of audio analyses.
After setup was complete, the company was able to run ongoing tests both during and outside of normal business hours. This means that nights and weekends—when most human testers would be loath to come into the office—the robots kept chugging along.
Ultimately, these tests uncovered a number of bugs that would have been hard to catch in a normal service verification process—and those bugs were sent back to human engineers who were able to pinpoint their locations and work to address them.
In this way, the company was able to stretch its resources to make ongoing regression possible.
The Benefits of Continuous Regression Testing
In the example above, SEGRON’s client was able to improve their standing a top telco operator in Europe with their increased ability to find and resolve bugs. Based on our experience, this is a fairly typical outcome for introducing automation into a regression testing environment.
We’ve alluded to this fact above, but, in general, automated continuous regression testing tends to have a few distinctive benefits for telco operators:
- Increased opportunities to dedicate network engineer competencies and skills to performing root analyses and taking steps to fix bugs;
- Major savings in terms of testing costs (due to the decreased need for ongoing human test resources);
- Improved test coverage and test quality (due to the unvarying, repeatable nature of automated tests);
- Shorter time-to-market.
If there’s an overarching theme to these benefits, it’s that automation for this use case gives you the flexibility to put your resources where they can do the most good. To the extent that you’re striving for higher test coverage, you’re working to automate in an area where automation can be especially impactful.
By the same token, by taking testing tasks off the plates of your engineers, you’re freeing up those resources to provide value elsewhere. It should be pretty easy to see how this would result in higher network quality overall, and thus better customer retention and an improved bottom line. It’s just a matter of letting the robots do their jobs.
