Friday, November 1, 2013

Black Tuesday - The technical side of things

Our goal at The Bruery is to create exciting beer and along with that, to share that beer with as many people as we can. It sucks when the plan to share the beer becomes an issue, as it did earlier this week with our Black Tuesday 2013 online sale.

Though our online store website initially failed us, it did eventually start working, and thanks to our amazing and patient fans we were then able to get Black Tuesday into the hands of more people than ever before. We know how frustrating this was, and we are very sorry. We appreciate the flood of comments offering advice and support, so we felt it appropriate to give you guys an outline of what happened.

As some of you may be aware, we switched to a new eCommerce system at the beginning of this year.

We had our share of issues throughout the year with stressed product launches, and worked closely with our vendor to address the problems. At a certain point it became clear that our vendor did not have the ability to provide what we needed on their systems, so we switched to a new company for both hosting and consulting.

They were able to provide a more scalable solution to add resources during planned high volume releases in order to address the problems that we had been experiencing. The original problem being faced was the ability to handle the sheer number of users attempting to access the site at the same time. By separating server roles, increasing memory and processor resources (by more than 10 times in some cases), and setting up a load balanced web cluster, we were able to eliminate this bottleneck.

Unfortunately, by fixing that problem it just introduced another problem which is the database. After extensive testing it became evident that the current shopping cart platform we are utilizing is not well suited for handling a large number of concurrent visitors, and there is not much that can be done to extend its functionality to do so. Adding more hardware resources proved to have little gain as the issue is not with the power of the servers but the efficiency and design of the code.

We are currently evaluating our options, including the cost to switch to a different platform that can handle the load, however no decision has been made yet.

Believe us when we say that we get no satisfaction out of seeing our site crash and our customers become upset. We certainly can't guarantee that we'll have a perfectly working site next year, but we can guarantee that we'll reexamine our current processes so that sharing our beer is easier in the future.

Until then, we hope you enjoy this year's release as much as we do!

8 comments:

Jeff Roth said...

Why not just use BPT? They don't seem to have these issues when they handle order brewery releases. Is it just too expensive to go through them?

Jeff Roth said...

Other, not order.

BuddyC Solutions said...

Thank you for a clear explanation, not only of the difficulties you encountered, but of the steps you are taking to remedy the situation. I appreciate that you have taken ownership of the problem and have offered an unqualified apology. I look forward to enjoying my purchase!

Susie Comi said...

I greatly appreciate the explanation... I knew what I was working toward, so I didn't give up!

Michael S Patterson said...

I have to agree with Jeff. Go with a service that specializes in high traffic/short time sales. All this time and energy spent on hammering out a way to sell high demand items online and not brewing seems ridiculous. Give up some control on eCommerce and keep up the high standard on brewing!

Jess said...

I appreciate all efforts made by you to bring me an awesome beer!! I waited and it came back online and I made my purchase . I had heard the stories about the crashing. I passed the Black Tuesday crash of 2013!!!!!! I will now spread the gospel of the terrible day when I had to sit at my computer and wait..HAHAHAHAHAHA

Jeff Roth said...

And it still wasn't as bad as the duck duck gooze release a few months back. So you guys are doing something right! Keep up the great work guys.

Dan Ucko said...

It was frustrating to see this happen directly after the stress testing, but I appreciate this post and the emails. Would've liked some responses on social media like I've seen in the past, but I'm sure you guys were flooded with inquiries good and bad. An improved ecommerce experience is definitely crucial -- but shipping to other places in California is a great start!