Friday, November 1, 2013
Black Tuesday - The technical side of things
Though our online store website initially failed us, it did eventually start working, and thanks to our amazing and patient fans we were then able to get Black Tuesday into the hands of more people than ever before. We know how frustrating this was, and we are very sorry. We appreciate the flood of comments offering advice and support, so we felt it appropriate to give you guys an outline of what happened.
As some of you may be aware, we switched to a new eCommerce system at the beginning of this year.
We had our share of issues throughout the year with stressed product launches, and worked closely with our vendor to address the problems. At a certain point it became clear that our vendor did not have the ability to provide what we needed on their systems, so we switched to a new company for both hosting and consulting.
They were able to provide a more scalable solution to add resources during planned high volume releases in order to address the problems that we had been experiencing. The original problem being faced was the ability to handle the sheer number of users attempting to access the site at the same time. By separating server roles, increasing memory and processor resources (by more than 10 times in some cases), and setting up a load balanced web cluster, we were able to eliminate this bottleneck.
Unfortunately, by fixing that problem it just introduced another problem which is the database. After extensive testing it became evident that the current shopping cart platform we are utilizing is not well suited for handling a large number of concurrent visitors, and there is not much that can be done to extend its functionality to do so. Adding more hardware resources proved to have little gain as the issue is not with the power of the servers but the efficiency and design of the code.
We are currently evaluating our options, including the cost to switch to a different platform that can handle the load, however no decision has been made yet.
Believe us when we say that we get no satisfaction out of seeing our site crash and our customers become upset. We certainly can't guarantee that we'll have a perfectly working site next year, but we can guarantee that we'll reexamine our current processes so that sharing our beer is easier in the future.
Until then, we hope you enjoy this year's release as much as we do!