Safely Bring up Your Network After a Power OutageRita Mailheau
As we’ve grown increasingly reliant on powered and PoE technology, power outages have become more disruptive than ever before.
We may have come a long way as it pertains to preventing them, but there’s still room for improvement. That being said, they will continue to happen on occasion, and it’s critical that the right organizations know how to quickly and effectively recover.
5 years ago, on September 8th 2011, a 500-kilovolt high-voltage transmission line from Arizona to California failed and left folks from San Diego to Arizona without power during a heatwave
A few years later, the San Diego Union Tribune reported that the outage began as the result of a botched procedure at a transmission switchyard outside Yuma. It was quickly followed by a painful “cascade of failures eventually knocking out the San Onofre nuclear power plant.”
A 2012 federal inquiry found that the outage resulted from “a lack of effective contingency planning and coordination, describing the outage as largely a consequence of human failures, and not hardware.”
While it’s fair to say nobody could have seen this coming, it definitely goes to show that network installers and operators need to be aware of the ongoing risk these types of events present.
Forward-looking IT operations and data centers that have already invested in implementing N+1 redundancies were able to weather the storm. But for everyone else, a reboot was in order.
Today, we’re going to cover how you can safely recover from a power outage and how to get everything back online quickly and safely.
Identify the Root Cause of Your Power Outage
Before you do anything else, it’s important that you determine why you lost power.
Unplanned outages can occur for a variety of reasons. San Diego Gas & Electric (SDGE) cites the most common reason for power outs as traffic accidents, damage to power lines, and storms and high winds.
On rare and unfortunate occasions however, a small-scale disaster may also be the root cause.
These could be events like localized fires. Assess the risks accordingly.
What you may not have realized is that local fires activate sprinkler systems. Since water conducts electricity, it should go without saying that these situations can be quite dangerous.
Similarly, gas leaks can also be involved on occasion. If this is the case, turning lights on can actually trigger an explosion.
So if you have an unexplained power outage on your hands, always be sure to consider the safety risks before turning anything back on. If there’s standing water or gas involved, it’s best to notify your utility company and involve trained professionals.
How to Power up Once Risks Have Been Assessed
Let’s get down to brass tacks. Once you and your team have decided it’s safe to proceed with powering up, what’s next?
Start by turning everything off. Switch all of your network device power switches to the off setting or unplug them. This protects system components and is the first prerequisite to controlling the power-up sequence.
Now, with everything unplugged, it’s safe to check your fuses and replace any that may be damaged or broken.
Wait at least 30 seconds if it’s just the circuits. This is to ensure that everything is completely turned off.
Now it’s safe to initiate the reboot sequence.
If you have pulled out any power cords, it’s now safe to reinsert them.
Turn on your managed switches. Confirm that each device properly boots and be sure to double-check that your network configurations haven’t been lost.
Your technician will need to upload a saved configuration or reconfigure the switch before allowing other computers access to the network. This is more for security purposes than connectivity.
Unsecure switches and routers would quickly grant access to any and all network devices connected or passing through them. Since limiting access points is a foundational element of network security best practices, any unsecured device needs to be isolated at very least.
Examples of this would include guest Wi-Fi networks, which are often found in manufacturing and commercial installations.
One by one, power on your routers. Be sure to wait for each router to fully boot up before moving on to the next one. Generally, you’ll know it’s done booting up when the lights stop “blinking”.
The last step to getting your network up and running is to turn on your modems. Again, go one by one so you can prevent the need to troubleshoot later if any of the devices were damaged during the power outage or surge.
Remember that order, and you’ll be all set. Once these three steps are completed, it is now safe to turn on end user devices such as computers and connect to the network.
Final Thoughts on Sequencing Device Bring Ups
Surprisingly there’s little written on “the rules” of how one should go about powering up after an outage. The whole point of following the guidelines above, is to ensure maximum network security and stability.
This logic also holds true when applied to server only networks. If anything, more is at stake in those circumstances since maintaining database integrity is generally the top priority for operators and technicians.
With any luck, you (a network installer) won’t have to find yourself in a scenario where you’ll need to put this knowledge to work. But we trust it’s been useful to help you further your understanding of network component protection and security procedures.
If you’d like to learn more, please feel free to give us a call. Our engineers would be happy to help you improve your network and grow your business.