The Pros and Cons of Keyword Based Testing

Let’s say you have a small system whose functionality you want to test, and you determine that there are five distinct use cases that require verification. As a technically-adept test engineer with a fair amount of programming skill, what do you do? In all likelihood, you simply script up each test case individually, so that for each use case there’s a unique piece of code that needs to run in order to make sure that everything is working smoothly. This makes sense, and as test automation has become more common (in some sectors, anyway), we’ve seen a lot more people doing just that. For a limited number of test cases, this is probably the smart thing to do. But what happens when it’s not half a dozen use cases that require scripting, but hundreds or thousands?

Needless to say, things could get cumbersome quickly. Not only would the actual scripting be extremely time consuming, but tracking the various use cases throughout your library could also become difficult over time—especially if you wound up with some duplicate test cases. Clearly, if you want to automate at scale, you’re going to need to adapt this paradigm a little bit. This is where keyword based testing comes in. But what exactly is keyword based testing, and how effectively does it add value compared to the alternative?

What is Keyword Based Testing?

Keyword based testing typically arises in test environments that have adopted at least some form of automation (it can work with manual tests in some software testing environments, but in the telecom world that may be less applicable), which might explain why the technique is becoming more popular in the present moment. Keyword based tests begin by dividing the testing process into two steps: test case design and test case implementation. In the design phase, rather than scripting up each test case individually with a unique piece of code, users can reuse existing code almost endlessly by swapping pre-defined keywords in and out.

The idea here is that test scripts would contain recognizable keywords in order to offer a test framework that provides test execution and documentation that’s relatively readable on an intuitive level. If, for instance, you were trying to test a call between PSTN subscriber and a mobile subscriber on an LTE network, you would define the two devices (already controlled by your test framework) as something like “pstn_a” and “ms_b.” From there, you would type in the relevant keyword (“Dial And Call” for instance) in order to define and execute the test action that the system should take. From there, the interface would let you know whether the call had succeeded or failed.

Crucially, the various use cases defined in your test library would all be helpfully tagged, such that testers could easily find all of the relevant test cases for, say, acceptance testing with a new device, or functional testing for some new network elements. Because the appropriate test cases can be grouped together so easily, testers can quickly run through relevant test cases and examine the results in the same keyword based environment. This means that if you’re looking for call failures in your various verification passes, you can simply search for them, and then seek out the relevant part of the system for performing bug fixes. Naturally, you can add and edit test cases as needed within the same framework, and even use the same scripts and libraries if you’re scaling or migrating your automation efforts.

Pros of Keyword Based Tests

Based on the way we described keyword based testing, a few benefits should be pretty obvious. Specifically, the readable and repeatable nature of the test framework reduces set-up time for all but the first round of tests and makes it possible for less technical personnel to be involved in the testing process. This is no small benefit. Because high levels of test coverage have been getting harder and harder to achieve in recent years (due to the influx of new devices, protocols, and technologies into the telecom world), anything you can do to ease the time and resource pressure on small groups of test engineers can radically speed up testing. Though the initial set up will require some scripting knowledge, the vast majority of the testing functionality won’t, meaning that users without any formal coding knowledge will still be able to hop in and run through some test cases. On top of that, there are a handful of other benefits to using this kind of approach:

• Easy integration of new devices and network into test cases as needed
• Reduced costs for test maintenance
• Quality UX with the potential to shave further time off of tests
• High degree of code reusability
• Easy access to test information
• Integration with external files

If there’s an overriding theme to these pros, it’s this: keyword based testing offers real flexibility, and that flexibility has numerous ways to add value for testing operations. If, for instance, you’re rolling out a new service offering (laying the groundwork for 5G adoption with new network elements, say), you don’t have to reinvent the wheel in order to run tests on the system for the first time. Likewise, you put yourself in a position to run regression tests on an ongoing basis as needed after the initial rollout is complete.

Limitations of the Keyword Based Method

Because we put the ‘pros’ first, it might seem like keyword based testing is a kind of test engineer’s panacea. Sadly, there is no such thing. As such, there are potential hurdles and limitations to keep in mind when considering implementing a keyword based framework. First of all, for the initial setup, it does require more knowledge of the system under test than, for instance, typical black box testing might—which potentially makes it more difficult to outsource. By the same token, a keyword based strategy also requires a fairly carefully managed environment in order to work: if you’re not able to track who should be doing what, when, then your test results could easily languish unaddressed. To that point, it’s important to organize your test suite into groups of related tests with clear objectives and goals, ideally with separate testing levels for things that do and don’t require user interaction.

All that being said, there are operational hurdles that need to be overcome in order to make any kind of automation possible—but there’s also the potential for a considerable return on investment. If you’re able to find an automation framework that fits in with your ideal version of service verification, keyword based tests can be a powerful tool for stretching your resources and improving the quality of your test reports.

LEARN MORE ABOUT TEST AUTOMATION

Are you looking for a solution to automate your service verification that includes both true end-to-end testing and full access to the systems under test?