Making the transition from manual to automated tests can be daunting. In many cases, specialized knowledge is required for almost every aspect of a tester’s job. At SEGRON, we know firsthand that anyone who’s thinking about adopting a new testing framework is going to have a lot of questions. Luckily, when it comes to our own ATF (Automated Test Framework), we’ve got answers.
Below, you’ll find a sampling of frequently asked questions about our testing framework. Not seeing your question on the list? Feel free to contact us through this link, and we’ll be happy to walk you through any concerns you might have.
The ATF is designed to be as user-friendly as possible, which means that scripting up test cases yourself is incredibly simple. We use a keyword-based scripting language based on Robot Framework, which means that even users with no development skills can write simple tests with a tiny bit of training—meaning that the whole range of ATF functionality is easily accessible across the entire organization.
What do ATF reports look like? How can we use them to evaluate test results and uncover root causes?
Every time the ATF runs, it automatically produces a report utilizing the same keywords referenced in the test scripts. At the top level of abstraction, you can see which tests were run, how long they took, and which tests passed/failed—and from there you can drill down into more granular information including protocol-level analysis in order to find the root causes.
These reports can be accessed from within your ATF dashboard, and because they’re keyword-based it’s easy for anyone in your organization to access, read, and understand them.
What about notifications? Can we implemented automated system notifications for specific failure scenarios?
Yes, you can absolutely receive automatic notifications in order to respond to incidents and failures as quickly as possible. Notifications can also be integrated in the network monitoring system as needed.
Does your framework require rooted Android devices or iOS devices that have been jailbroken in order to run tests on them?
ABSOLUTELY NOT! Our framework orchestrates testing using only out-of-the-box devices—exactly the devices that your subscribers will be using to access your network. In so doing, we avoid all the risks that come from testing on rooted devices: security issues, loss of device integrity, and inaccurate test results.
Click here to learn more about the risks of testing on rooted.
About 5 minutes. All it takes is a subscription to SEGRON iDA (our AI based device automation product). iDA stands for intelligent Device Automation, which is a device learning used to automate the integration of new mobile devices into the test environment as soon as they come on the market. The learning process identifies the necessary device characteristics and generates a configuration file that is sent to the testing framework, enabling it to run tests on the new device.
What other devices can be incorporated into the test flows? Can the ATF orchestrate testing with tools like load generators and simulators? What about fixed line (PSTN) phones?
You bet! In terms of testing tools, if there’s an API we can use it to orchestrate testing. This is due in part to the magic of Robot Framework, which offers sophisticated tools for process automation.
On top of that, we can automate fixed line phones, fax connections, and IP phones in addition to a wide variety of mobile phones. This can obviously be crucial for testing out legacy interworking and other similar functionality.
What about apps? Can you automate Facebook Messenger, WhatsApp, and other apps that subscribers use frequently?
Yes we can. It’s basically the same logic as mentioned above—if it’s got an API, we can incorporate it into an ATF environment as needed, as shown in the video below.
Indeed it can. The short version of how we accomplish this is, basically, that the ATF controls both the relevant end-user devices and digital attenuators in order to simulate a waning LTE signal and force a fallback or handover. This effectively reproduces the conditions under which subscribers moving around your network (or testers doing the same) might run into LTE dead zones. In this way, you can cut out the lengthy drive tests that might otherwise slow down service verification and adversely impact Time-to-market.
Yes, the ATF Remote Framework can orchestrate tests from a central location, easily managing devices in different locations and countries. Using the central ATF Hub, controlling the remote ATF instances enables a holistic view on service quality, roaming and fraud detection on a global basis, as presented below.
Yes, the ATF can capture signalling traces in order to provide protocol-level information to testers about the system under test. This is the basis of the ATF’s “Beyond End-2-End” functionality. In this way, testers can verify the actual protocols on a message level sent by their phones or network elements and check the results against what the signalling data should have revealed. This means that you can get a much more accurate, in-depth picture of what’s happening on your network, and when bugs arise you can find the root causes that much more quickly.
Learn more about going Beyond End2End here.
Yes, it can. All test suites are conducted on the timeline that you specify, whether that means recurring executions every hour, regression tests performed at night while the system isn’t being used by engineers, or around the clock network monitoring. Not only does this potentially speed up time-to-market, it can help boost your testing ROI through increased flexibility and improved time efficiency.