Sites Outage this Morning & How We Recovered
Today was a shockingly challenging day. We woke to discover that websites were down. We thought it was just one website, but we soon found that multiple websites were effected; possibly all of them!
This website design video is what went wrong, how we responded, and what we’re doing to prevent the same thing happening in the future!
Website Outage crisis averted! Thanks, Picard!
What Went Wrong
We use a hosting provider who manages each website install. One of our web design clients is in a legal dispute over trademark. The site hosting provider was supposed to “take down” just one website, but instead accidentally took down the entire block of websites.
How We Responded
When we discovered that multiple websites were down as of 7:45 a.m., we checked multiple websites and found that there were more than a few that were “hard down”; not able to pull up on any device.
We immediately got our hosting service on the phone, as well as their infrastructure company. (There are multiple layers of website infrastructure, and you usually have to dive deep into each layer to find what went wrong.)
We then rolled through each of our normal redundancies:
- Upload and activate one of our 24 hour website backups of each website. (No success.)
- Upload and install the backup on another hosting service, such as moving from BlueHost to GoDaddy (No success.)
- Purchase an additional “block” of websites and one by one install and re-launch them. (No success.)
- Manually install and connect the SQL databases, which is time-consuming, but a last-ditch effort. (No success.)
- MEANWHILE, we were still working with the Support Reps with our main hosting service.
Finally, after several hours of these attempts, a plucky Support Rep got to the root of the problem; a trademark infringement lawsuit with one particular website. He appealed to an underlying infrastructure vendor to re-activate the “block” of websites while the lawsuit was still being settled.
It turns out, it was not a technical failure in the least! It was one of those 1% probability situations where a legal issue created a technical issue.
The block of websites were instantly re-activated and we notified the affected site owners.
What We’ll Do to Prevent Future Website Outage
In the immediately future, the Spark Sites team is doing the following MULTIPLE redundancies:
- Installing our own WordPress Backup plugin on every website to have access to our own backups (not wonky vendor backups.)
- Install our own monitoring plugin to notify us the moment a website is down.
- Create an internal policy to partition any website that is known to have legal issues (even if that website client is in a settlement, to be sure other websites are safe.) We will not shut down the website of concern. We will instead move it to another host, and point the domain immediately.
That is how we will ensure that LEGAL issues do not create TECHNICAL issues for your website in the future!
Thank you for your continued business and support, folks!
Spark Sites
Lakeland Website Design and WordPress Support Services
www.SparkMySite.com
