The road to scalability: from infrastructure mess to success

Over the course of the past year my startup, KYA and its analytics product, has undergone massive improvements in both features and functionality. But before we were able to build out shiny new features, we had to address the elephant in the room: scalability.

It was pretty much exactly a year ago that I brought on my CTO, Joel, who is incredible, and he essentially sat me down and told me (and I’m paraphrasing here): “The infrastructure is a mess, you can’t bring on any really big sites, and it’s going to take about 6 months to get KYA fully scalable.”

…Not what I wanted to hear, as you can imagine.

While it wasn’t what I wanted to hear, I needed to hear it, because of course, Joel was absolutely right. We had experienced so many issues with our old infrastructure that it was impossible to work with. We would add something new and subsequently break something else. It was a vicious cycle and on top of that, it was not scalable at all, whatsoever. I prayed every night before I went to bed that I wouldn’t wake up to the world burning.

The next 6 months were spent getting better logging implemented so we could more easily identify errors and bugs, moving our code from a cluster of Digital Ocean nodes to Heroku, moving from Sinatra to Rails, moving away (mostly) from Postgres to using AWS (Redshift, Kinesis Firehose), and just spending a lot of time cleaning up code and eliminating technical debt.

If you’ve never done any of this before, let me tell you, it is a long process–especially because you want to do it right this time around to avoid major headaches later on. And as a founder, it sucks. It sucks because you want to be signing on big customers (but you’re hamstrung by your infrastructure). It sucks because you need to devote all of your resources to infrastructure migration and pretty much none to new feature development. It sucks because you end up feeling frustrated because it seems like the product has stagnated (even though it really hasn’t–solid infrastructure is crucial to a good product).

Nonetheless, 6 months later we had a scalable product and it felt (hell, it still does) incredible. And guess what? Since July/August 2016 we have been able to build out amazing new features at an impressive pace and nothing has broken. We even were able to build an iOS app and Google Chrome extension that will be launching soon.

Bottom line: the road to scalability is long, and at times, hard and annoying, but remember: in the end it will be so, so worth it.

Here are some key takeaways I picked up from the experience:

Reliable products are critically important, perhaps even more so for products like KYA which collect data that people rely on to be correct to make strategy decisions. Scalability and reliability go hand in hand. If your product isn’t reliable it doesn’t matter how cool it is if people can’t use/access it.

Listen to your CTO when (s)he tells you not to sign on a big customer unless you want everything to topple over. And remember: first impressions are important.

Scalability isn’t like turning on/off a light. It doesn’t happen with the flick of a switch. It’s a methodical, planned, process that takes place in stages over a period of time. This means you will notice things actually start getting better as each stage is completed, however, don’t mistake this for true scalability (until you’re 100% done).

Once you have a scalable product, the sky is the limit. It’s a really beautiful thing.

Have any thoughts on building a scalable product? Leave a comment!