The Top 5 Causes of Network Outages

As  mobile voice and data services have become a communication necessity for business and private purposes,  network outages will continue to be a serious problem for telco operators. A Heavy Reading report from a few years ago suggested that network outages in the telco domain seemed to be getting both longer and more frequent, with a rise in the volume of outages that exceeded 48 hours.

Though network conditions, and the networks themselves, have changed since then (it was still relatively early in the transition of IP replacing other protocols for real-time services), the possibility of an outage is still top of mind for most operators. After all, outages can do serious damage to your corporate reputation and result in subscriber churn and revenue losses.

At SEGRON, we believe that the first step to preventing network outages is understanding their primary root causes. The second step… well, that depends on which cause was actually responsible for the outage in the first place.

1. Overloaded Networks

In the report we mentioned above, congestion and overloaded networks were listed as the top causes of outages. On one hand, the timing of this report means that part of what we were seeing several years ago was networks grappling with the shift away from mostly voice and SMS traffic to increasingly diverse network traffic that emphasized low-latency mobile data and VoLTE usage.

On the other hand, the industry is currently entering a new era of transition from 4G/LTE to 5G. You can now expect another sustained increase in network traffic, this time from the internet of things (IoT). As 5G powers an increasing volume of low-latency network connections going forward, load testing will be more crucial than ever for telco testers.

In order to ascertain whether your network can actually hold up during periods of congestion, you’ll have to find practicable ways to stress test the network itself. This will require automation frameworks that are capable of leveraging entire networks of connected devices at the push of a button.

2. New Services

While network congestion was cited as the most frequent cause of outages, network issues were the leading culprits when it came to serious and severe outages. These issues can take any number of forms, but some of the most common were outages related to new service offerings.

Again, this is something that the 5G era is likely to see a lot of, for example:

You’re upgrading your network so that subscribers can access 5G data speeds when streaming video, but something goes awry with    the implementation and the network crashes.

 You’re updating your roaming partnerships and the new interconnections cause issues that you didn’t anticipate.

The traditional bulwark against this kind of issue is extensive regression testing, but that can be time-consuming and slow down time-to-market, which companies are often hesitant to do. The trick here is finding a way to speed up regression testing to improve test coverage without impacting time-to-market.

3. New Devices

For network failures, new service offerings are more frequently cited than the introduction of new devices, but the latter still causes its share of outages. At the risk of sounding like a broken record, this is another area where the risk is only going to increase as mobile networks evolve and more devices enter into common usage.

To decrease the odds that a new device will cause issues for your network, you’ll need to implement a robust testing infrastructure for devices that includes both end-to-end tests (dialing and calling, sending SMS messages, using mobile data, etc.) and tests that go beyond end-to-end.

What does this mean, exactly? Essentially, it means gathering signalling traces and tracking EDRs/CDRs from the systems under test. This allows you to get protocol-level information about how each element of each test is being carried out. From there, you can identify potential issues that aren’t yet manifesting themselves at either endpoint.

The only trick here is incorporating newly released, flagship devices into your automation framework quickly enough to maintain test velocity, but AI-powered workflows can be a big help here.

4. Misconfigurations

Another frequently cited cause of network outages is misconfiguration, which can typically be translated to human error. In the telecommunications industry, there’s a lot of manual effort that goes into any new service offerings, changes to your network, or adjustments to the equipment you’re using. While there’s no way around the fact that “to err is human,” it is possible to decrease the amount of manual intervention that goes into any of these updates and changes.

You can start with ramping up automation in your test labs by creating test scripts for each use case that do the same thing, the same way, every time without deviation. From there, it will produce test reports the same way every time, leading to greater transparency between testing and other functions.

Obviously there’s no way to root out human error completely, but you can detect it and reduce it through tactics like this.

5. Physical Failures

This is another term that works as something of a catch-all. Obviously, a physical failure can be the result of anything from a hurricane taking down phone lines to rats gnawing through the wiring of your network equipment. Some of these may be partially human error again, but often times they really will be situations that are out of your control.

That said, there’s plenty you can do to prepare your organization for things like this when they do happen. For instance, any infrastructure you can put in place to help you identify the issue more quickly will be a huge advantage. This can be a matter of testing, but it’s also important to retain documentation in a way that’s accessible for key stakeholders.

For instance, if someone has identified a similar issue in the past, it’s of the utmost importance that whoever’s responding to the outage can locate, access, and understand that fix and other similar fixes. Reporting matters; it’s crucial that your incident report be readable and consistent, one of the many benefits of keyword-based testing. From there, you’ll also want an official operational plan for any possible outage scenario, outlining who’s in charge of doing what and when.

Of course, there’s no way to stop outages completely, but an ounce of prevention is worth a pound of cure. If you are eager to know more about how SEGRON can help you mitigate the costs of network outages, contact us.

Search

Interested in our Products ?

Scroll to Top
Segron logo black blue

Senior SaaS System Administrator

Technical Skills :
  • Oversee the sysadmin related tasks in our SaaS infrastructure (partially cloud based, partially bare metal)
  • Daily operation and maintenance of the system
  • Analysing and resolving incidents
  • Follow and help improving the incident and change management procedures
  • Design procedures for system troubleshooting and maintenance
  • Incorporating base OS updates and security patches
  • Ensure that systems are safe and secure against cybersecurity threats by raising change requests where potential threat is possible
  • Performing SW updates for the Segron SaaS SW stack (distributed architecture with clusters)
  • Configuring solutions like reverse proxy, firewalls, etc.
  • Building tools to automate procedures & reduce occurrences of errors and improve customer experience
  • Tutoring & coaching newcomers & less senior experts in the team
  • Interworking with the architects and IT admins of Segron to have the SaaS procedures inline with the Segron processes
Non-technical skills:
  • We are looking for a self-motivated, self-improving individual with a highly independent mindset and open and straightforward technical communication to help us to improve and maintain our cloud infrastructure of our powerful end-to-end testing solution ATF (Automated Testing Framework)
  • 3+ years hands-on experience with operation and monitoring of cloud / linux systems
  • 3+ years of hands-on experience with network devops elements: configuring routers, switches, networks
  • Hands-on experience with running live systems with infrastructure as a code mode of operation
  • Specific knowledge which brings direct advantage: Docker, Docker Compose, Grafana, Prometheus, Ansible, Debian Linux OS administration, Security
  • Experience in building and maintaining distributed systems (incl. redundancy, resiliency, load-balancing) is welcome
  • Excellent knowledge of English
Location :
  • Place of work: Bratislava (partially home office possible)
  • Rate: from 30 EUR/hour (possible higher rate, depends on experience)
Segron logo - The Next Generation of Active Testing
Segron logo black blue

CI/CD Senior Developer

Technical Skills :
  • A senior role with a proven expertise in software development, cloud computing, DevOps, and CI/CD
  • Experience in planning, designing, and overseeing the CI/CD strategy and architecture on the level of organization
  • Ability to tailor testing strategies which define and follow the best practices, standards, and policies for the software delivery process
  • Hands-on experience in creating and managing CI/CD pipelines and workflows (PaaC)
  • Ability to evaluate and recommend the best tools, technologies, and methodologies for the CI/CD implementation
  • Prior hands-on experience working with different CI/CD toolsets (Jenkins, Bitbucket, GitLab, artifactory, Ansible ..)
  • Proficient with DevOps tools API automation capabilities
  • Proficient with Atlassian Tools (BitBucket, Jira, Confluence) and agile SW development methodologies
  • Familiar with cloud patterns and best practices
  • Familiar with web performance best practices
  • Comfortable working in cloud DevOps ecosystem
  • Comfortable working with Linux platforms
  • Initial working experience in SW development is an advantage.
Non-technical skills:
  • Effective communication with technical as well and business stakeholders
  • Self-motivating, self-improving mindset
  • Ownership of relevant industry certificates is a plus
Location :
  • Location: Bratislava, Slovakia (with hybrid flexibility)
  • Rate: from 30 EUR/hour (possible higher rate, depends on experience)
Segron logo - The Next Generation of Active Testing
Segron logo black blue

Test Automation Engineer

Job description, responsibilities:

Hardware Testing: Conduct verification and testing of hardware during the HW/SW products
assembly process.
Technical Support: Provide technical assistance during testing activities and troubleshoot any
hardware issues that arise. Support to internal teams and customers by managing priorities and
meeting deadlines.
Equipment Maintenance: Configure and maintain SEGRON laboratory and customer equipment.
Logistics Support: Handle hardware ordering and logistics.
Troubleshooting: Analyze and resolve SEGRON product issues through effective problem-solving.
Ability to troubleshoot and resolve issues that arise during the integration process.
Collaboration: Work with the SEGRON Development Team on product or software issues and assist
the Project Team with planned activities.
Root Cause Analysis: Investigate production errors to identify underlying causes.
Software Deployment: Implement software updates and fixes.
Team Collaboration: Ability to work collaboratively in a team environment, especially in serving global
customers.

Requirements/ Skills:

Networking knowledge: Understanding TCP/IP networks and LAN/WAN configurations. Routing and
switching and related protocols strongly desired.
Hardware Proficiency: Understanding of hardware components, including servers, routers, and
peripherals, to ensure proper assembly and integration. Experience in data centers is a plus.
Travel Abilities: Willing to travel for customer on-site support for hardware installations, migrations,
upgrades, troubleshooting, etc. (few times a year, 3-5 days)
Hand Tools: Proficiency in using hand tools like screwdrivers, pliers, crimpers, and punches for
assembling and disassembling components.
Operating Systems: Knowledge of Linux and Windows. VM experience (Hyper-V, VMWare, Oracle
Virtual Box, Proxmox etc.)
Tools Proficiency: Basic understanding of Ansible.
Telecommunications Experience: Previous experience in the telecom sector is a plus.
Education: Degree in Computer Science/Engineering or equivalent work experience.
Experience: At least 2 years of relevant work experience preferred. A strong desire to continuously
improve skills and knowledge in hardware and software integration, with an openness to feedback and
new ideas. Key is openminded person willing to learn fast in great team.
Communication Skills: Effective oral and written Eng

Others:

• Full time job (employment)
• Onsite work during onboarding period, then it can be 3 days onsite, 2 days home office
Offered salary: from 2000 Euro (depends on seniority and skills level)
Variety of financial benefits
Place of work: Bratislava

Segron logo - The Next Generation of Active Testing
Segron logo black blue

Senior Python Developer

Technical Skills :

  • A solid, experienced SW developer with at least 10 years of experience in active SW development in different programming paradigms
  • Minimum 5 years of professional Python development experience
  • Master or college degree from Computer Science, Mathematics or STEM domain
  • Well educated in design and programming patterns that increase software’s efficiency and readability.
  • Very good analytical and problem solving skills.
  • At least three skills out of the following 4 skills are requested:
    • Microservices based architectures (Docker containers)
    • Linux
    • Ansible
    • Robot  Framework
  • Comfortable with sysadmin and DevOps skills (Ansible, YML files, Network Programming, IP protocols, designing and developing proxy servers for different protocols – example: streaming, integrating and compiling third party libraries on Linux (Debian))
  • Proficient with Atlassian Tools (BitBucket, Jira, Confluence) thorough understanding of Git and version control best practices
  • Familiar with cloud patterns and best practices
  • Familiar with web performance best practices

Non-technical skills:

  • Ability to work under pressure
  • Ability to abstract and explain your work
  • Strong understanding of Agile development process and experience working in an agile team
  • Strong communication skills with both technical and non-technical stakeholders
Location :
  • Bratislava, Slovakia (with hybrid flexibility)
  • Rate: from 35 EUR/hour (possible higher rate, but depends on experience)
Segron logo - The Next Generation of Active Testing