Making the transition from manual to automated tests can be daunting—particularly in the telecom domain, where so much specialized knowledge is required for almost every aspect of a tester’s job. At SEGRON, we know firsthand that anyone who’s thinking about adopting a new testing framework is going to have a lot of questions. Luckily, when it comes to our own ATF (Automated Test Framework), we’ve got answers.
Below, you’ll find a sampling of frequently asked questions about our testing framework. Not seeing your question on the list? Feel free to contact us through this link, and we’ll be happy to walk you through any concerns you might have.
1. How complicated is the test scripting process? Can we write test scripts ourselves?
Answer: The ATF is designed to be as user-friendly as possible, which means that scripting up testcases yourself is incredibly simple. We use a keyword-based scripting language based on Robot Framework, which means that even users with no development skills can write simple tests with a tiny bit of training—meaning that the whole range of ATF functionality is easily accessible across the entire organization.
Learn more about Test Case scripting here.
2. What do ATF reports look like? How can we use them to evaluate test results and uncover root causes?
Answer: Every time the ATF runs, it automatically produces a report utilizing the same keywords referenced in the test scripts. At the top level of abstraction, you can see which tests were run, how long they took, and which tests passed/failed—and from there you can drill down into more granular information including protocol-level analysis in order to find the root causes.
These reports can be accessed from within your ATF dashboard, and because they’re keyword-based it’s easy for anyone in your organization to access, read, and understand them.
Learn more about reporting here.
3. What about notifications? Can we implemented automated system notifications for specific failure scenarios?
Answer: Yes, you can absolutely receive automatic notifications in order to respond to incidents and failures as quickly as possible. Notifications can also be integrated in the network monitoring system as needed.
4. Does your framework require rooted Android devices or iOS devices that have been jailbroken in order to run tests on them?
Answer: ABSOLUTELY NOT! Our framework orchestrates testing using only out-of-the-box devices—exactly the devices that your subscribers will be using to access your network. In so doing, we avoid all the risks that come from testing on rooted devices: security issues, loss of device integrity, and inaccurate test results.
Click here to learn more about the risks of testing on rooted.
5. Okay, but what about newer flagship devices? How long does it take to incorporate them into the framework?
Answer: About 5 minutes. All it takes is a subscription to SEGRON iDA (our AI based device automation product).
6. What other devices can be incorporated into the test flows? Can you orchestrate testing with tools like load generators and simulators? What about fixed line (PSTN) phones?
Answer: You bet! In terms of testing tools, if there’s an API we can use it to orchestrate testing. This is due in part to the magic of Robot Framework, which offers sophisticated tools for process automation.
On top of that, we can automate fixed line phones, fax connections, and IP phones in addition to a wide variety of mobile phones. This can obviously be crucial for testing out legacy interworking and other similar functionality.
7. What about apps? Can you automate Facebook Messenger, WhatsApp, and other apps that subscribers use frequently?
Answer: Yes we can. It’s basically the same logic as you saw above—if it’s got an API, we can incorporate it into an ATF environment as needed.
8. Can the ATF automate mobility-related use cases like SRVCC, handovers, etc.?
Answer: Indeed it can. The short version of how we accomplish this is, basically, that the ATF controls both the relevant end-user devices and digital attenuators in order to simulate a waning LTE signal and force a fallback or handover. This effectively reproduces the conditions under which subscribers moving around your network (or testers doing the same) might run into LTE dead zones. In this way, you can cut out the lengthy drive tests that might otherwise slow down service verification and adversely impact time-to-market.
9. Can the ATF handle devices spread across multiple locations?
Answer: Yes, the ATF can orchestrate tests from a central location, easily managing devices in different locations or even other countries. In point of fact, this can be useful functionality when it comes to verifying roaming service.
10. Can you analyze signalling traces with the ATF?
Answer: Yes, the ATF can capture signalling traces in order to provide protocol-level information to testers about the system under test. This is the basis of the ATF’s “Beyond End-2-End” functionality. In this way, testers can verify the actual protocols on a message level sent by their phones or network elements and check the results against what the signalling data should have revealed. This means that you can get a much more accurate, in-depth picture of what’s happening on your network, and when bugs arise you can find the root causes that much more quickly.
Learn more about going Beyond End2End here.
11. Can your system execute tests 24/7?
Answer: Unlike human testers, the ATF never gets tired, never needs a break, and never deviates from the established way of doing things. This means that all test suites are conducted on the timeline that you specify, whether that means recurring executions every hour, regression tests performed at night while the system isn’t being used by engineers, or around the clock network monitoring. Not only does this potentially speed up time-to-market, it can help boost your testing ROI through increased flexibility and improved time efficiency.