July 28, 2021
It’s good practice to always prepare for the unexpected, but as the Northern Hemisphere heads into summer and severe weather like extreme heat, fires, and hurricanes threaten an increase in power outages, it is critical that data center operators stay ready. That starts with a business continuity and disaster recovery plan that eliminates a common cause of data center downtime: human error. A well-curated checklist reduces the risk of mistakes and oversights in the midst of a disaster and ensures an organization is prepared for any eventuality.
- Risk assessment: This should be the first step for any organization in developing a disaster recovery plan. What types of threats are relevant to your area? Hurricanes, flooding, tornadoes, fires, earthquakes, and volcanoes are examples of natural disasters that require planning, but are any of your facilities located near areas that make radiation exposure, toxic waste, or explosives a planning consideration?
- Evacuation plan: Human safety always comes first, so you should have a plan to evacuate any personnel potentially at risk. This should include a detailed plan for communication with staff to confirm their safety.
- Weatherproof the data center: If the threat is a hurricane, flooding, or something weather-related, take the necessary steps to harden your facility. Secure or store loose items and make sure servers are secured in their racks. Clear gutters and storm drains. Make sure doors can be sealed against high winds and blowing rain. Water is the enemy of the data center, so do everything necessary to ensure no water enters the server rooms.
- Backup data: Many data centers conduct routine data backups once a week. If you know severe weather is coming, increase the frequency of those backups. We can’t always know when a disaster is going to strike, so organizations should consider making daily backups a regular practice. Consider where data is being backed up. It should go off site, but make sure the off-site location is secure and safe from the potential disaster.
- Check the generator: There is a tendency to set and forget a generator, but that piece of machinery requires maintenance and upkeep to ensure it performs as expected when needed. Is it full of clean fuel? Are the fuel line and air filter free of contaminants? Test the generator regularly and ahead of any anticipated weather events. Line up at least three vendors to deliver fuel in the event of an extended outage. Remember, fuel often is at a premium after a disaster, and yours will not be the only organization requiring delivery.
- Communicate with utilities: Consider the ramifications of loss of power, water, phone, or internet. Communicate early with utility providers to set up contingency plans. Create a contact list and have a plan for communicating if traditional channels are compromised.
- Emergency staffing: In the event of a significant disaster, local employees may be unavailable to work. They may have evacuated with their families, be dealing with urgent damages to their homes or vehicles, or be unable to reach the data center due to impassable roads. Consider bringing in emergency crews and establishing crisis housing near the data center to ensure you have on-site personnel.
- Contact vendors: Establish a list of vendors and prioritize those requiring communication in the event of an emergency. Reach out to them early and make the necessary arrangements so you can be free to focus on more immediate needs during the crisis.
- Trust your team: Bring together all parties — IT, Facilities, Security, HR, Communications, Legal, Logistics, Information Security — and make sure everyone understands their responsibilities throughout the crisis. Have a plan for communicating with that team in the event the usual communication channels are down.
- Confirm insurance coverage: This starts with insurance on the facility itself, but additional coverage may be warranted on the infrastructure or for continuity of business. If the data center is down for a week, continuity of business insurance can compensate the organization for lost revenue.
- Remember the edge: Today, the enterprise data center is just one piece in a distributed network. Many organizations manage multiple edge sites. Those sites are more critical than ever before, and they must be considered in disaster planning. In many cases, the core data center may be safe from a specific event, but one or more edge sites could be at risk. Prioritize by criticality. Have a plan for those facilities and any personnel at those sites.
- Mind the cloud: Just because some of your data and applications are housed in the cloud, does not mean they are always safe from emergency events. Those cloud servers are in a data center somewhere, and you should know how your cloud provider will handle a potential disaster. How often are they backing up data? Do they have redundant sites? Ask these questions before a crisis, because once disaster strikes, it’s too late.
- Consider the opportunists: Hackers see natural disasters or similar emergency events as an opportunity to access networks while attention is focused elsewhere. Make sure your information security and physical security teams are prepared for bad actors.
Having a data center disaster recovery checklist is an invaluable part of business continuity planning — a collaborative process with your team that should lead to regular plan updates, especially as changes are made to critical equipment and personnel.
Not only can the right data center services provider be a part of that team, helping with the risk assessment needed to properly prepare for a disaster, but these infrastructure experts are specifically trained to ensure a timely and safe recovery. Contact a service provider near you for help with your planning.