Website Outage this Morning & How We Recovered

Sites Outage this Morning & How We Recovered

Today was a shockingly challenging day. We woke to discover that websites were down. We thought it was just one website, but we soon found that multiple websites were effected; possibly all of them!

This website design video is what went wrong, how we responded, and what we’re doing to prevent the same thing happening in the future!

Website Outage crisis averted! Thanks, Picard!

Website Outage Crisis Averted! - Thanks! - Lakeland Website Design - SparkMySite.com

What Went Wrong

We use a hosting provider who manages each website install. One of our web design clients is in a legal dispute over trademark. The site hosting provider was supposed to “take down” just one website, but instead accidentally took down the entire block of websites.

How We Responded

When we discovered that multiple websites were down as of 7:45 a.m., we checked multiple websites and found that there were more than a few that were “hard down”; not able to pull up on any device.

We immediately got our hosting service on the phone, as well as their infrastructure company. (There are multiple layers of website infrastructure, and you usually have to dive deep into each layer to find what went wrong.)

We then rolled through each of our normal redundancies:

  1. Upload and activate one of our 24 hour website backups of each website. (No success.)
  2. Upload and install the backup on another hosting service, such as moving from BlueHost to GoDaddy (No success.)
  3. Purchase an additional “block” of websites and one by one install and re-launch them. (No success.)
  4. Manually install and connect the SQL databases, which is time-consuming, but a last-ditch effort. (No success.)
  5. MEANWHILE, we were still working with the Support Reps with our main hosting service.

Finally, after several hours of these attempts, a plucky Support Rep got to the root of the problem; a trademark infringement lawsuit with one particular website. He appealed to an underlying infrastructure vendor to re-activate the “block” of websites while the lawsuit was still being settled.

It turns out, it was not a technical failure in the least! It was one of those 1% probability situations where a legal issue created a technical issue.

The block of websites were instantly re-activated and we notified the affected site owners.

What We’ll Do to Prevent Future Website Outage

In the immediately future, the Spark Sites team is doing the following MULTIPLE redundancies:

  1. Installing our own WordPress Backup plugin on every website to have access to our own backups (not wonky vendor backups.)
  2. Install our own monitoring plugin to notify us the moment a website is down.
  3. Create an internal policy to partition any website that is known to have legal issues (even if that website client is in a settlement, to be sure other websites are safe.) We will not shut down the website of concern. We will instead move it to another host, and point the domain immediately.

That is how we will ensure that LEGAL issues do not create TECHNICAL issues for your website in the future!

Thank you for your continued business and support, folks!

Spark Sites
Lakeland Website Design and WordPress Support Services
www.SparkMySite.com

recent posts

The Idea Behind Designing ADA Websites for Accessibility

The Idea Behind Designing ADA Websites for Accessibility

Understanding the concept behind making websites accessible: To understand why we are designing ADA websites, visualize your life without the internet for a moment. It's not about a shortage of the router or the supplier, it means life without an Internet, or life...

Thorough Design is Good Design

Thorough Design is Good Design

Last week in our post we touched on the troubles of website trends through Dieter Rams' 7th principle of design: good design is long-lasting. Today we want to really break apart what we think is one of the most important principles drawn out by Mr. Rams. This post is...

Long-Lasting Design is Good Design

Long-Lasting Design is Good Design

Last week our post was about Dieter Rams' 6th principle of design: good design is honest. This week we're diving into his 7th: good design is long-lasting. What does this mean, for something to be "long-lasting?" In the ever changing realm of technology it can be...