Earlier this week I wrote a guest post for the Google App Engine Blog on how Pulse has scaled-up our backend infrastructure to prepare for the recent Kindle Fire launch.
The Kindle Fire includes Pulse as one of the only preloaded apps and is projected to sell over five million units this quarter alone. This meant we had to prepare for nearly doubling our user-base in a very short period. We also needed to be ready for spikes in load due to press events and the holiday season.
Livecount is an implementation of real-time counters that leverages the performance of memcache and task queues on Google AppEngine.
Building a solid analytics platform is often a combination of real-time and batch processing. Batch processing, with a tool like Hadoop, is great for digging into large amounts of past data and asking questions that cannot be anticipated. In contrast, when it is known ahead of time that certain aggregates will be required, the best solution is usually to count each event as it happens. Livecount makes it easier to address this second use-case.
A few months ago I mentioned that I left the government/research world (Sandia Labs) and joined an exciting new startup. I’d like to share a bit more about my experience so far and announce that we are hiring!
Those who have worked at a large company and then moved to startup can probably relate to my experience. First, without a doubt, the most motivating and fun part about working at Pulse is seeing the impact of my work. And I don’t mean just having someone say “Good Job” or receiving a strong performance review, I mean seeing thousands of people USE the results of your work and submit feedback about how it benefitted their lives. At Pulse, this experience is magnified by the fact that we release new product features every two weeks, and not ever quarter, or every year!
Google AppEngine is a great tool for building simple web applications which are automatically scalable. All of the basic building blocks are readily available and accessible from both python and java. This includes a database, a caching layer, and support for background tasks.
What about the big data analytics and informatics that made Google famous? Does AppEngine help us there as well? The answer is yes; although with some serious limitations.
After a year and half of big data research for the government and quite a bit of fun with Hadoop, I’ve decided to join some good friends at an early-stage startup called Alphonso Labs.
Pulse is currently the #1 news reader on the iPad, iPhone, Andriod app stores. I’ll be leading the development of our backend data platform and working with a great team.
As we start to build out Pulse’s backend, I’ll be continuing to experiment with Google App engine. Stay tuned for more posts in that regard.
A few thoughts from my first test of Google App Engine.
My goal was to put up a prototype java web app for pushing email alerts based on RSS content (more to come on the full idea). Unfortunately, it took much longer than I expected to get things going (longer than my web app protoype took to write) – leaving me feeling a bit disappointed. On the up side, the app has been running quite well for about a week now.