In my previous post I told the story of how the idea of Volafile came to life. This post depicts how, seven months later, I started working on what was someday to become Volafile.
Someday, seven months ago
Staring blankly at my screen I wondered what to work on. I had just
finished an Android app and had no project to hack away on now.
Out of the mass of unfinished projects and ideas I had lying about, I was trying to pick something decent. That torrent search engine? No. The android wlan sniffer? What about the realtime filesharing site? That kind of project might fit me just fine right now…
I chose Node.js for the job, mostly because it was hip at the time and I always like to try something new. I also picked mongodb and added redis to the mix. As mongodb already came with a built-in ‘cloud’ filesystem I began to build everything with GridFS as my storage backend.
As (not) expected, it performed poorly. The first and pretty obvious problem was the poor performance of GridFS. I was barely able to write more than 5mb/s into it, reads weren’t really fast either. After looking into hundreds of different kinds of ‘Cloud Filesystems’, ‘Distributed Object Storages’ and the like, I decided to roll my own solution. What I came up with was able to use the drive throughput of my servers to the fullest, while scaling horizontally (lie, but with a little bit more work it will) and being written in a few hundred lines of node.js. Eventually I completely eliminated the dependency on mongodb (I just read about why it was supposed to be bad on HN anyways) and moved the whole database into redis.
So Volafile was now running on a nginx, node.js, socket.io and redis stack, most basic trickery working on the client side (albeit with a much simpler, and more ugly UI than today) and I felt it was time to put this thing online. I registered the domain volafile.org, mentioned it in a few places around the internet and waited for people to come. They came, and with them the performance problems became obvious. They became even worse when Volafile ended up being on the frontpage of Hacker News for about 12 hours. Thousands of people were sent to the page and unable to use it because Volafile was barely able to handle the load.
It wasn’t the servers. This shouldn’t happen.
Something was obviously very wrong.
To be continued in part 3.