Dear liveBooks Clients,
The fact of the situation is that we're in uncharted territory. After seven years of having a 99.997% uptime record we have hit a rough day. We can only imagine how frustrating it has been for you. I am deeply sorry.
Some good news is that you can access email via your webmail account (webmail.yourdomain.com).
As for your websites, you would expect from a company such as ours, redundancy is very important, which is why we have redundancy in our systems at every stage we can – from the load balancers at the front-door of our data center right down to the databases that store all of our customer and website data. There are a few places, however, in the networking infrastructure (the pipes that connect it all together) where complete and fail-proof redundancy is more difficult, and in some cases, not practical to achieve.
It was one of these pieces of the networking infrastructure that failed today. Our monitoring systems caught this immediately and our systems engineers have been working on this since moments after it first occurred. We've had our most senior systems engineer physically at the data center since very early this morning. Everyone at liveBooks has had their full attention on resolving this issue today. It has been all hands on deck to help in every way we can.
We are upset that this issue isn't resolved yet and all of us are still working full-time on resolving it. The sites have come up a few times today for short periods of time and then have gone back down. Some of you have understandably asked for an ETA. We are doing our best to resolve this issue as fast as we can. We will let you know as soon as the issue is resolved and we will do our best to explain in detail what happened. We will do this via the User Form ( http://forum.livebooks.com/livebooks ), the support phone line (1-888-833-4605) and social media.
We have taken into consideration your suggestions. For instance we have updated our support phone line with information about the outage and will continue to do so, as well as utilizing Facebook and Twitter to stay in touch. We have made over 500 outbound phone calls today to inform you of the outage and sent some fifteen thousand emails.
Although I do appreciate many of your suggestions as to what has happened, the fact of the matter is that the network is a highly complex system of many moving parts. Some of you have have written posts or tweeted that you 'found a DNS and suggest that we have re-routed to LA" or that "we have taken our own corporate site in priority above the customer's sites." I can tell you that nothing is farther from the truth as your team at liveBooks is fully dedicated to you as a business partner with integrity and transparency as values we hold dear. I am proud of the team here and the efforts they have put forth to assure this is resolved as soon as possible.
We'll be back in touch soon.
Andy Patrick | President & CEO | liveBooks Inc.
Help get this topic noticed by sharing it on Twitter, Facebook, or email.
Good morning to most of you. We had a few intermittent issues over the night, each lasting a few minutes. All sites are up and running now, as is email and editSuite, and continue to stabilize. Don't hesitate to contact firstname.lastname@example.org if you see anything funky. It is a new day and I hope it is a great one for you all. Much gratitude.
EMPLOYEE0Dear liveBooks Clients,
Next week I still plan to send you a detailed report as to what happened this past Wednesday and the numerous steps we are taking to improve. In the mean time we feel that it is necessary to perform a hardware configuration update. Specifically, we are going to do a BIOS update to one of our servers.
This update requires intentional downtime of our network. This will take place tomorrow, Saturday June 23rd at 12 Midnight PST and ending by Sunday June 24th at 2:00 am PST. Your liveBooks website and editSuite may not be available during this two-hour time period. Your email should continue to work during this time.
Based on your suggestions this week, we have created a liveBooks Status site on Tumblr’s servers where we will update you regularly on the status our liveBooks network. Please note the site also contains links to the many ways that we can communicate with each other.
With continued gratitude and respect,
President & CEO