Let’s say you’re working through a whole slate of different test scenarios to verify service on a new network that you’re rolling out. One of your first tasks is tackling VoIP (voice over IP) tests, which, as it turns out, present some very particular challenges.
Because jitter and latency in voice conversations can quickly render a call frustrating and incomprehensible, your tests have to seek out extremely granular data about packet loss and packet delay for a number of different use cases. In order to do so effectively, testers need a wealth of specialized knowledge.
Where less time sensitive forms of communication tend, to a certain extent, to maximize packet size in order to reduce overhead, the nature of human speech means that VoIP protocols tend to prefer smaller (64 byte) packets, such that if a single packet is lost the listener will only miss a fraction of a syllable.At anything under about 3% packet loss, your quality of service won’t take much of a hit. For packet delay, anything over 200 ms is likely to cause quality of service issues, while for jitter (the variation in delay time among packets), that number is more like 35 ms.
Because these numbers leave so little room for error, they require not just specialized knowledge but frequently also specialized equipment. Why? Because virtual test environments are often just dissimilar enough from reality that a few milliseconds here or there can slip through the cracks.
Add this to the fact that use cases that require testing are proliferating every year, and it’s no wonder more and more telco operators are opting to outsource their testing. The question is: how risky is it to do so?
Hurdles for In-house Testers
We covered this somewhat above, but let’s quickly lay out the argument in favor of outsourcing tests. First of all, testing has become more complex and time consuming than ever, with new protocols constantly popping up while legacy telephony continues to require verification.
A human tester can run through about 6 use cases per day, meaning that reaching high levels of in-house test coverage often seems daunting or impossible. At the same time, customer standards for quality of service are getting more and more stringent, meaning that low test coverage isn’t an option. Things can’t stay as they are—something’s gotta give.
By outsourcing tests, telco operators solve these immediate problems. All of a sudden it’s feasible to achieve high test coverage and work towards improved network quality without taking up the time of a small army of in-house engineers who would otherwise be working on more valuable projects.
In addressing the immediate issues in this way, however, telecoms can set themselves up for additional hurdles down the road. Why? Because most test outsourcing options don’t empower their clients to retain institutional testing knowledge, making it harder and harder to regain control of one’s tests even as new technologies like automation make in-house testing more feasible
Institutional Domain Knowledge
In the first section above, we threw out a handful of numbers for acceptable latency times and rates of packet loss. If you’re in the business of testing VoIP with any regularity, you likely have those numbers or similar ones in mind already, just as you know off-hand a host of other granular details about how to establish your test environment, which use cases to verify, and what to look for in your tests.
If you’re not in the habit of performing these tests regularly, on the other hand, you probably don’t know most of these details, and would need to rely on outside resources in order to effectively verify service on your own. When this happens at a personal scale, it’s not necessarily impossible to recover—but when it happens at an institutional scale, the results can be worrisome.
If, as a company, you find that after a few update cycles with outsourced tests you’ve lost the ability to do the work in-house, even on a small scale, you essentially put yourself in a position where you’re totally reliant on your testing provider.
This reduces your agility as a company and increases the potential for organizational disconnect and the emergence of information silos. Though your test coverage may be much higher than it would have been otherwise, you might not be able to maximize the value of that test coverage because your internal processes don’t align exactly with the way your testing partner presents their findings.
And since your team potentially can’t reproduce the test conditions, it may be difficult to actually fix any bugs that have been uncovered.
Alternate Paths
Okay, so when you outsource testing you risk losing operational agility and domain knowledge, which can result in ineffective bug response and reduced test quality in the long run. Does this mean that telco operators are stuck devoting almost unthinkable time and resources to reaching high levels of in-house test coverage manually? Thankfully, the answer is ‘no.’
Though the risks we outline above do exist, it is possible to partner with a testing provider who can help you maintain and even improve your institutional domain knowledge, rather than letting it languish.
By providing robust and comprehensive reporting that clearly defines its use cases and KPIs, a testing provider can ensure that you have exactly the information you need to reproduce and understand test results. Since tests are only truly valuable when they’re repeatable, this ought to be a deal breaker.
If we’re talking about a service that can run automated tests (which would make test coverage a non-issue), they might even be able to give you the tools you need to run automated tests in-house, thereby sidestepping the entire problem.
In this way, you can boost your use cases per day up into the hundreds, saving resources and person hours for other tasks like actually resolving service quality issues. All of sudden, your agility is restored and you have ownership over your own service verification operations.
This means fewer budget surprises and more oversight from within your corporate structure—to say nothing of the increase in practical knowledge throughout the organization.
