Scaling Pulse to 11M Users

Written in 2012 after Pulse crossed the 11 million user mark. I was the founding engineer and backend lead.

As part of Pulse’s recent announcement of crossing the 11M user mark (up 10x since last year!), we’ve written a set of blog posts to share how we’ve scaled our backend infrastructure to keep up with our new users and support some powerful new features. Here’s a quick recap of our systems on both Amazon Web Services (AWS) and Google App Engine (GAE), along with links to the detailed posts describing each.

AWS has been a great platform for building systems as close to the metal as we need to, while still avoiding the burden and inelasticity of owning the hardware. At Pulse we use AWS extensively for event logging, data analytics, curating top story feeds, recommendations, monitoring, etc.

In addition to our infrastructure on AWS, we’ve built a very scalable serving infrastructure on GAE. This infrastructure makes it possible for us to improve the mobile reading experience of millions of users by serving extremely optimized content, usually in under 100ms. It also hosts our Pulse.me web app and the API that allows users to sync sources, save stories, update their profiles, and share stories with shortened urls.

Leveraging and combining the strengths of these two platforms (and the systems we’ve built on them) allow us to provide the best possible experience to all of our current users and continues to enable us to scale rapidly in support of our continuing growth.