How Can Telco Operators Speed Up Conformance Testing?

When technology changes and evolves—as it does almost constantly—in the telecom domain, it typically takes standard-setting bodies like 3GPP six months to a year to establish a new set of test cases for conformance testers.

Once those test cases come out, there’s a flurry of activity while operators, device manufacturers, OEMs, and others attempt to verify compliance and interoperability with new and existing standards.

The fact that it takes 3GPP a fairly long stretch of time does very little to lessen the time pressure that testers usually face when it comes to performing each new round of service verification.

What are testers supposed to do under these conditions? Everyone wants faster conformance testing in order to ensure high network quality, but given the increasing complexity of the networks, devices, and protocols under test.

It seems like there’s virtually no way for manual testers to even maintain their current pace, let alone speed things up. If that’s the case (and we think that it is), then it’s time for the telco domain to begin moving away from manual conformance tests and towards real test automation.

Why? Because it presents testers with their best past forward for running through an increasing number of use cases to verify service.

Hurdles in Conformance Testing

Because conformance tests tend to be based, at least in part, on standards developed by outside organizations, there’s often a bit more uncertainty baked into the process than usual.

Rather than either reusing test cases or developing new ones in tandem with the developers making changes or updates to your network, you’re stuck waiting for the starting gun to go off and then adapting 3GPP’s recommendations within the context of your own LTE network, for instance.

This means that your testing velocity is based not just on how many use cases you can run through in a day, but also how quickly you can configure those test cases based on the information you’re receiving.

This will require understanding the standards themselves and breaking down each element into a set of levels and modules that can be mapped out and visualized.

The more easily you can do that, the more quickly you can get your tests underway. Even at this point, however, you’re struggling against the challenges that face all varieties of telecom testing right now—namely that increasing complexity is driving down the number of use cases a human tester can verify in a single day.

This number, on average, is down from about 10 last decade to more like 6 today. As a result, even if you establish your testing framework quickly, it’s still difficult to turn around the tests themselves in a timely manner.

The Power of Automation

For operators seeking a new approach, this is where automation comes in. First, on the level of test case design, any automation solution worth its salt will have an easy-to-use scripting language for defining use cases.

This means that when your test cases are relatively unknown during the lead-up to a test, you can still get things off the ground fairly quickly. Once the testing begins, you can boost your test output from half a dozen test cases per engineer per day to hundreds or thousands of test cases.

Broadly speaking, as more and more protocols need to be tested for compliance and interoperability, test coverage threatens to go down—but by ditching manual tests and letting an automated test framework take over the actual verification stages, you can reverse that trend, improving your test coverage in record time.

It’s worth noting here that when we talk about automated tests we don’t mean simulations. As new technologies like 5G arise that have increasingly precise standards and service quality expectations, the differences between real, out-of-the-box devices and equipment and simulated versions of the equipment will become increasingly noticeable.

This might not seem like much when you’re simply checking to see whether a call was completed successfully or not, but even when it comes to functional testing the line between success and failure can gain a certain fuzziness that testers need to fight against. In the long run, failure to identify, document, and resolve issues that may crop up in conformance testing is one of the other key factors that slow down this kind of verification.

Why? Because the accumulated issues gum up the works of your test lab and engineering teams, making it difficult to maintain agility overall. In an automated environment, this loss of agility can be staved off through faster test execution with higher coverage.

Achieving ROI

Okay, we’ve seen the reasons why conformance testing in the modern telco environment can be slow and painful, and some of the ways in which automation can help. But are faster conformance tests an adequate end in and of themselves, or should they be powering some larger goal like faster time-to-market or improved ROI?

From our perspective, it’s absolutely the latter. Why? Because whether your tests are automated or manual, it’s still far too common to see poor reporting or confusion over roles devalue test results.

Finding gaps in functionality is all well good, but how useful is it if uncovering those issues doesn’t result in the issues being resolved in a timely manner?

In this way, we can reimagine our standard for test velocity to cover not just the speed of the tests themselves, but how quickly and effectively the reporting translates into results. If your tests, automated or not, uncover an error on SIM password validation for certain equipment.

But your engineers can’t figure that out based on the test’s output (whether because it’s hard to read, hard to access, or stuck in an information silo), you’re potentially achieving and verifying compliance more slowly than you would be otherwise. Here, automated reporting has the potential to offer a massive advantage.

But only if the automation vendor you select makes reporting transparency a priority. If they do, you stand to increase not just test velocity but actual service verification velocity for your conformance tests, improving your time to market and freeing up valuable engineer time in the process.