Today’s post is by Tech Support Specialist Chris Martin.
Here’s a cautionary tale about staffing and software from our own experience…
Hurricane Irene was coming. We didn’t know how badly it was going to affect Virginia Beach and the rest of the Hampton Roads area. It could take a turn for the worse and hit us directly, which was unlikely. The storm could hit during high tide and seriously flood the area, which is already prone to flooding. Or, it could be like any number of tropical storms that come through here where the residents continue on with indifference.
The days leading up generated a casual interest and by Friday, the day before the storm would hit, we still did not know anything more. The wind was stronger, the waves on the beach had the surfers excited and the rain was unrelenting. Emergency preparations for the offices were planned and being implemented.
Our sister staffing company, Reliance Staffing & Recruiting, made their own preparations for the hurricane to keep operations going as well as possible.
In case of power outages, the offices were printing out an Availability List for those employees who had called for whatever kind of work recently. This report included the employee’s phone numbers if they needed to be contacted for any impending work or, in a real disaster, to jump on recovery contracts. They also printed out their active job orders with the job orders’ employee list to have on hand. In the event of a power outage, the office staff would continue to be a communications hub for the needs and scheduling of their employees and clients.
The internal staff also unplugged their computers, setting them on top of their desks in case of flooding, and ensured the windows were closed and locked and that the shades were drawn if they had any.
The corporate office had a little more to do, though. SQL Server and COATSsql were installed on a laptop. We did a manual backup of the data and placed a copy onto that same laptop. Having been through hurricanes here, we knew it was a possibility for some areas to have power outages for a week or two or more. Therefore, it was necessary to be prepared to print employee checks from a laptop, plugging it and the printer into a backup generator if needed, though unlikely.
We did local backups in addition to ensuring the regular backup ran successfully before leaving. Everyone had everyone else’s phone number and a call chain was established for the internal staff. The main voicemail for both companies was updated for the circumstances.
As the storm was being monitored, eventually it was decided to power down the servers Saturday morning as a precaution. Fortunately, power was never lost at the office and the damage to Virginia Beach was minimal.
However… Monday morning, the SQL Server would not boot. Four of the 6 drives failed on the server. Reliance had placed an order for a new SQL Server a couple months prior, but the order had been delayed and they still did not have the new server to replace this. We had to get creative, but we managed to get Reliance back up and running in COATSsql by 11:30 that day. It was a frantic morning restoring from the backup and creating an impromptu server.
We had felt good about our preparations, and this wasn’t our first hurricane, so we were unpleasantly surprised by the server problem. Was the situation preventable? Possibly. You never know what can happen, so it is important to be as thorough as possible in your preparations.
Many government agencies consider it regular maintenance to shut down their servers once a quarter, then boot them back up. Doing so in Reliance’s case may have prevented the drives from failing at an inopportune time after so many years.
Now it’s hurricane season again. Do you have a plan to protect your offices, your employees and your data?
To get more of this insight and expertise, call Chris at 1-800-888-5894 or email him.