City Loses All Critical Information—Electronic and Paper—to a Fire

A small city with two servers also stored many paper documents containing critical information. The city backed up its servers with tape-based data backup which a city employee would take home every week or so to store “offsite” at their house. Many of the paper documents were not replicated electronically, and so these paper documents were the only versions in existence.

One night, a fire began that destroyed nearly all the building before firefighters arrived at the blaze. Fire alarms went off but no fire suppression occurred until the fire department showed up.

Assessing the damage the next morning, the city discovered that its paper documents and servers were completely destroyed. With the paper a total loss, the city decided to recover the server data from the tape backups. However, after a two-day attempt at trying to restore the data, the city could only retrieve about 10% of it. Many of the tape backups hadn’t been tested and the city didn’t realize that the backups weren’t running properly for a long time.

As a result, operations ground to a halt and the city found itself in dire trouble. They lost their accounting and billing systems along with many public records and documents. So many critical operational records were lost related to accounting, taxpayers, and businesses. The public outcry had only yet to begin after the admission of data lost—and why the city had not properly backed that data up.

Preventing This Disaster

A fire can happen to any city at any time. Is your city prepared? For such a common disaster, we find that many cities do not have disaster recovery plans that account for a simple yet deadly fire.

Let’s look at the errors in the story above.

Error #1: Using paper as the only copy of important documents.

In today’s electronic information age, relying only on paper for important documents is way too risky. A simple fire can wipe out paper in a matter of minutes. Paper also fails in a flood, tornado, or other natural disaster. Any paper-based documents that are critical to your city need to be scanned electronically and backed up offsite to ensure they are not lost.

Error #2: Poor offsite data backup plan in place.

Relying on a city employee to take tapes offsite every week to their house is not a sure-fire plan. First, these tapes were not tested on a regular basis. When the city actually needed to restore data, most of the tapes failed. Second, too many security and liability risks exist when a city relies on an employee to manually collect backup tapes and store them in a private home. What happens if the employee is negligent or disgruntled? What if they forget one week to take the backups home?

Error #3: Lack of appropriate fire suppression for a server room.

Any room that stores servers needs best-of-breed fire suppression. Fire alarms alone are inadequate. Most data centers feature fire suppression technology that helps eliminate or reduce the severity of a fire. If your city decides to host its own servers, then you need to explore fire suppression options beyond an alarm.

Error #4: Lack of an overall disaster recovery plan.

The city clearly did not think through the consequences of a disaster. Otherwise, it would have identified critical information—such as its paper documents—and planned for a worst-case scenario such as a fire. This plan would include:

• Identifying which data is most critical and cannot be lost.
• Estimating the maximum amount of acceptable downtime before restoring city operations.
• Detailing how the city will get up and running after a disaster.
• Outlining contingency plans while the data is being restored.
• Ensuring that any data backups are tested regularly.

While large disasters like tornados can seem more improbable and less likely, cities need to keep in mind that disasters also include more common scenarios like fires. A fire can wipe out critical information quickly. Your disaster recovery plan needs to account for both paper-based and electronic information—ensuring that you can recover your most critical information soon after a fire or other common disaster.

Questions about your city’s ability to protect and recover your most important information after a fire? Reach out to us today.

Original Date: 3/14/2017