Likely, we’ve all been there---looking for that important file on your computer only realizing we’ve deleted it by mistake. But Marco Marsala took this common accident to a whole new level.
He said he runs a small web hosting company, managing websites for over 1,500 clients. Last week he ran a piece of code to perform some automated tasks on his servers and, by mistake, included a bungled piece of code that wiped out all the data of his systems, including his clients’ data, according to his post on ServerFault tech forum, where he was asking for help.
The liable code was “rm -rf.” It basically told the computer running a Linux operating system to delete everything and don’t ask any questions.
What Marsala seems to have been aiming for was to target the command at a specific location that was supposed to be wiped. But “due to a bug in the code above this line” the command wasn’t told what specifically to delete—so it deleted everything.
But the worst was yet to come.
Companies usually have a backup system in place---specifically for recovery from situations like this.
Unfortunately, Marsala was running backup as part of the same automated operation. Thus, the system was still connected to the backup storage and obliterated its contents as well.
“If you really don’t have any backups I am sorry to say but you just nuked your entire company,” user André Borie commented on Marsala’s post.
User Michael Hampton even suggested Marsala should forget about rescue efforts and move on to the next step: “You’re going out of business. You don’t need technical advice, you need to call your lawyer.”
Others, however, offered some hope.
A top-rated response from user Journeyman Geek suggested Marsala should contact his clients, inform them of the disaster, and prepare them for a substantial downtime of his services.
Then he should hire more manpower and start to carefully, piece by piece, run a data recovery program on his systems. Another course of action would be to hand his hard drives to a data recovery company. But that may be “extremely expensive,” another user warned.
Multiple users pointed out Marsala had a flawed backup policy to begin with. The main problem was that all of his backups were still connected to his system. Common advice for companies is to regularly create copies of backup files on an offline storage media and place them offsite, perhaps in a safe box, so nobody can access them and they won’t be destroyed even if your main offices burn to the ground.
