How To: Regression Testing for MVNO

MVNOs (mobile virtual network operators) sometimes get a bad rap. This isn’t entirely without reason: A recent report found that MVNO download speeds across the U.S. are 23% worse than those of their host networks on average.

While download speeds for MVNO often could reach the same heights as their host networks, they tended to offer those speeds less consistently, leading to lower “consistent quality scores” in terms of meeting minimum speeds and staying below allowable levels of jitter, latency, and packet loss.

On some level, there is a reason for this. Host networks tend to prioritize their own traffic over that of their MVNO partners, and they often have better options for managing traffic out in the field.

At the same time, nowhere is it written in stone that MVNOs have to provide less consistent service—after all, customer retention is just as important no matter the technical specifications of your particular service offering.

For MVNOs, a lot of the quality of service (and thus customer retention) game can come down to testing. A robust service verification workflow is the only way to answer that all important question: is your network offering the functionality that it should in terms of both quality and functionality, i.e. billing, provisioning, user experience, etc.?

Today, we’ll go over an approach to MVNO testing (MVNO regression testing in particular) that gives testers in this domain the tools they need to effectively maintain a high network quality.

How MVNO Works

For starters, let’s quickly go over what MNVO is, and how it differs from a traditional network. Essentially, MVNOs buy excess network capacity from traditional MNOs in bulk and resell that capacity to end-users.

Typically, MVNOs will leverage the spectrum bands and base stations of their host network, but handle the SIM cards for users’ handsets themselves—usually offering prepaid wireless subscriptions to their customers.

Since you’re not maintaining most of the actual network equipment, this kind of business model has less overhead than your average MNO deals with on an annual basis.

The same logic naturally applies to testing: testing of basic network elements and equipment will likely be the purview of the host network, leaving MVNO testers to focus specifically on the end-user experience they’re providing for subscribers.

Challenges in MVNO Testing

The end-user expectations for prepaid mobile data subscriptions are often going to be fairly different from those of typical 4G/LTE subscribers.

Where a mobile subscriber on a monthly plan doesn’t want (and generally won’t seek out) much direct interaction with their carrier outside of invoicing, prepaid subscribers will often need top ups and other services on a semi-regular basis for as long as they hold onto their plan.

This increases the number of opportunities for something to go wrong in your UX or your OSS/BSS systems—and it also increases the impact of any error or bug in those systems.

If a user trying to add minutes or data to their plan finds that they can’t get the interface to function properly, they’re going to be deeply irritated—and they might decide that it’s better to pick a new MVNO rather than topping up their existing plan.

Because of these factors, there are three primary challenges for testers in this area:

  • User interface optimization
  • More frequent and more impactful subscriber provisioning/invoicing management
  • Revenue assurance

Of course, end-to-end tests for billing systems can be complex, as can functional testing for customer-facing applications. You need to make sure that the web interface is updating successfully each time your backend system updates, and that the two systems always show the same accurate, up-to-date information.

By the same token, each call or data session needs to deduct the correct amount from the subscriber account, in order to make sure that no one is getting more or less than they paid for. Again, the updated information needs to be reflected both in the backend system and the subscriber interface to avoid any confusion.

The Keys to Effective Regression Tests

The challenges we outlined above make MVNO testing a somewhat unique use case. And, of course, everything we outlined above needs to be verified in regression tests every time there’s a change in your network or the host network.

Since you may or may not have much visibility into your host network’s schedule for updates, patches, new offerings, and other network changes, speed is of the essence for maintaining high quality of service. As such, automation is a best practice for effective MVNO regression tests.

In an automated environment, you could remotely control both the end-user interface and the network provider’s internal CRM. From there, an end-to-end testcase might look something like this:

  • The automation framework performs a balance top up.
  • The top up is verified in the MVNO’s internal system.
  • The system checks to make sure the new balance is reflected in the customer interface.
  • The balance in the customer interface is verified against both the expected value after topping up and the number shown in the backend system.
  • From there, the system initiates a call or a data session to ensure that the defined amount is being deducted from the subscriber account.
  • The subscriber account’s balance is again verified against both the expected figure and the figure being