Case Study: Circuit Switched Core Network Automation

Let’s say you’re planning to migrate to a new high-redundancy CS core network—what’s the first challenge that comes to mind? For many of you, it might be network architecture. The structure of modern telco networks is increasingly complex, and any new network rollout is going to be much more involved as a result. To wit, you’ve got to make sure that you’re setting up the core network protocols correctly (in accordance with the 3GPP standards), that your network can interconnect successfully with other networks, that it has functioning access layers for end users devices, and much more. 

Once this functionality has all been mapped out, however, you need to verify that it actually does what you think it does. Given how many interconnected layers go into a complex service offering like this, it’s only natural that not everything will work exactly as expected on the first go-around—which makes comprehensive network testing that much more important. It also, unfortunately, makes comprehensive testing that much more time-consuming. Network operators have to find a way to strike a balance between test coverage and time-to-market—a balance that’s becoming more and more difficult to achieve with manual testing.

The Challenge: Increasing Complexity in Telco Networks

Recently, one of SEGRON’s clients was in exactly the position described above: they had decided to implement a state-of-the-art, high-redundancy circuit switched core network, and they had an aggressive timeline for rolling out that network. Before they could roll it out, however, they had to verify more than 500 complex, Priority-1 testcases. This was poised to be incredibly time-consuming for exactly the reasons we outlined above. The introduction of new devices, protocols, and standards that happens on a yearly basis in the telco domain meant that not only were there more tests that required verification, the tests themselves were more complex.

Up to that point, our client had partnered with another testing service provider who still verified service by hand. Outsourcing in this way helped ease what would have otherwise been a nearly-insurmountable testing burden on the telco operator itself—but their testing service could only work through about 6 tests per tester per day. Based on our experience, this is a fairly typical pace; if they had kept their testing in-house, their test throughput likely would have been similar. Unfortunately, at this rate 500 testcases would take more than 80 person-days to complete—which would have meant significant delays in terms of time-to-market. Sacrificing test coverage for a sped-up timeline could have had significant repercussions in the short and long term, but verifying every layer of core network functionality by hand was simply not feasible.

The Solution: Future-proof Telco Test Automation Software

To deal with the mountain of complex testcases—covering everything from device acceptance to mobility management to legacy interworking—our client decided to make the transition from manual to automated testing. They opted to make use of our ATF (Automated Test Framework) in order speed up verification by removing manual intervention. Notably, many of the competing test automation solutions under consideration offered automated testing on rooted devices—i.e. devices that had been hacked to allow the installation of testing software. This can help speed up testing, but at the expense of test quality. Why? Because by hacking mobile devices you potentially change them just enough that they no longer offer an accurate representation of how users will connect to your network. Luckily, with SEGRON’s solution that wasn’t an issue. We were able to orchestrate testing using out-of-the-box Android devices that were identical to those their subscribers would be using.

To be more specific, we utilized 20 out-of-the-box Android phones, situated within a complex RAN environment that empowered verification for 2G, 3G, and 4G connectivity. Within this testing environment, we offered both libraries of pre-scripted telco use cases that could be imported directly into the testing framework and the use of a keyword-based testcase scripting language based on Robot Framework. In this way, we made it easy for our telco operator client to incorporate all of the necessary tests into a comprehensive test suite that covered all 500 use cases. With the framework installed, the devices connected, and use cases transformed into test scripts, it was just a matter of typing in a few keywords and waiting for the magic to happen.

The Results: 95% Reduction in Testing Time

Following the initial test setup, SEGRON ATF was able to power through all 500 use cases requiring verification within about 10 hours. Considering that the whole suite of tests was expected to take 80 person-days when managed by hand, this represented a 95% reduction in total testing time. Not only did this put them in a position to speed up time-to-market and keep to the original rollout schedule, it also set them up nicely for regression tests. Previously, running regression tests after any network updates or changes would have been comparably time-consuming to the whole manual testing rigmarole—whereas now they could perform regression tests in a matter of hours. Not only would that help them when it came time to make future updates to their core network, it also gave them additional peace of mind in the period leading to the go-live decision.

Thus, not only did they accomplish the on-time rollout of the new core network, they also set up themselves up for future testing successes. Because the testcases were all keyword-based, they could be modified and reused for new use cases on an ad hoc basis, just by adjusting the keywords in a universal file. More than that, those same keywords were used to generate all of the test reports—meaning that they were more consistent and easier to read than what most manual test workflows actually produce. This meant that when gaps in service and other errors cropped up, they could find the root causes and address them that much more quickly. In this way, the operational benefits of an automated, future-proof testing infrastructure were poised to cascade into other operational areas, bringing additional benefits beyond saved time.