Even the mighty are vulnerable to human error
Posted by posted by Francis @ 1/31/2009 03:03:00 PM
This morning, while I was shopping at the store, I was looking for product reviews on my blackberry. Every search I made on Google issued a warning about the site being potentially harmful to my computer. Even sites that I know or run had this issue. This was obviously a mistake on the part of Google. They issued an explanation about this error on their blog.Basically, it was a manipulation error in the lists they manage of known bad sites. I am sure Google has procedures to ensure this doesn't happen but things like that still happen in spite of best efforts.
We fell victim to something very similar this week. While we were testing an upgrade to the software of one of our customer's web sites, we misconfigured the staging server and sent outdated notifications to a few hundred of their customers. This must have surprised the recipients and surely triggered a few customer service calls that they were not counting on.
What make this story interesting is that we had planned for this and had prepared a procedure for staging that made sure to neuter the notification service. A manipulation error in the server configuration caused all this planning to go to waste. After we had fixed our mistake and notified the customer to expect a few strange calls, we looked at what happened. It was clear that the mistake did not sit with one particular individual but was a failure of our internal communication for this staging procedure.
At least we learned something and we can fix our procedure. But today's Google issue makes me feel a little better about it. Even the biggest web company in the world has its bad days.
Labels: work

