Earlier this week I wrote a guest post for the Google App Engine Blog on how Pulse has scaled-up our backend infrastructure to prepare for the recent Kindle Fire launch.

The Kindle Fire includes Pulse as one of the only preloaded apps and is projected to sell over five million units this quarter alone. This meant we had to prepare for nearly doubling our user-base in a very short period. We also needed to be ready for spikes in load due to press events and the holiday season

To learn more about our architecture on Google App Engine, how we dealt with the recent App Engine pricing changes, and how we prepared for an expected increase of 5M+ users, check out the original post. You can also find the original post on Pulse’s Engineering Blog and some additional analysis from Derrick Harris on GigaOM.

Building a solid analytics platform is often a combination of real-time and batch processing. Batch processing, with a tool like Hadoop, is great for digging into large amounts of past data and asking questions that cannot be anticipated.  In contrast, when it is known ahead of time that certain aggregates will be required, the best solution is usually to count each event as it happens. Livecount makes it easier to address this second use-case.

I encourage you to read about our experience with Livecount at Pulse.

Data Persistance

Livecount initially stores all counts in memcache.  To minimize the risk of data loss, each time a count is updated Livecount creates a worker task to write that count from the memcache to the datastore in the background. If the count is ever evicted, it is reloaded from the datastore on the next read or write.

from livecount.counter import load_and_increment_counter

load_and_increment_counter(name=url)

from livecount.counter import PeriodType, load_and_increment_counter

load_and_increment_counter(name=url, period=datetime.now(), \
period_types=[PeriodType.DAY, PeriodType.WEEK], namespace="starred", delta=1)

Warning

Before I start, let me recommend that you don’t try this. The potential savings you gain from using pre-tax Commuter Check cards likely won’t be worth the pain of actually trying to buy something with them. Return the cards to your employer and ask them to enroll in another option for funding your commute costs pre-tax!

Update: The Autoload program via Clipper works great. Instead of buying a pass in person with a commuter check card, you tag on/off once at the beginning of the month to load a new pass.

Goal Prototyped Below

Use two Commuter Check cards issued by an employer (each containing $100) to purchase a zone 1-3 monthly Caltrain pass on a Clipper card (for $179).

What Not To Do (Because It Doesn’t Work)

This section is intended to lower your expectations to the appropriate level. Almost all advice I have received on this topic has been either been out of date or blatantly incorrect, resulting in more wasted time than I ever thought possible. Please take my own advice with a grain of salt, as anything related to Caltrain or Commuter Cards is likely to change in unpredictable ways in the near future.

Walgreens

Since monthly Caltrain passes are usually purchased at Walgreens, this is a common first stop when trying to achieve the goal listed above. Unfortunately, although Commuter Check cards are labeled as credit card compatible, they dont work in the standard machines and cannot be accepted by Walgreens. This fact does not stop most Caltrain/Clipper/Commuter Check employees and random people on the street from suggesting this option every time you ask.

Online / Over the Phone

For some reason, probably related to the same reason Walgreens can’t accept them, Clipper’s online system does not accept Commuter Check cards. Again, this fact does not stop Clipper/Commuter Check call center employees from suggesting this option every time you ask. The Clipper call center will even try to process your Commuter Check card and tell you there is something wrong with it. This will result in you sepending another 20 minutes on hold with Commuter Check customer service just to be told that the card is fine and that you should purchase your monthly pass online (see beginning of this section).

In Person

Explaining all of this to a Clipper call center employee will sometimes result in them telling you that you can only use your Commuter Check cards at a Caltrain booth with a clerk (real person). In my case, they strongly suggested I go to the 4th/King Caltrain station before 7pm. While this advice is getting closer to the right answer, if you actually try going to 4th/King and asking around for a clerk or a booth, you’ll eventually discover that all of the Caltrain clerks in the entire system were laid off a few months ago. The remaining employees at the station fall back to the commonly issued advice above (see Walgreens & Online / Over the Phone).

What Does Work (Sort Of)

I’d like to credit the clerk at the Walgreens across from the 4th/King Caltrain station with the first halfway useful advice. Apparently, the Transit Store at the Powell street BART station accepts Commuter Check cards in exchange for Caltrain monthly passes!

BART Transit Store

Amazingly, the magazine stand style booths at some of the major BART stations along market do take Commuter Check cards. Unfortunately, some open late / close early and asking a BART employee for their hours results in very inaccurate information. Also, for some reason the Civic Center booth (where I was advised to go because it opens earlier than most others), does not take Commuter Check cards (anymore). Where can you use them? Based on my experience, I know for sure that you can use them at the Embarcadero BART station Transit Store booth and should be able to use them at the Powell street station if you can get there when they are open.

Bad Cards

When I finally reached a person with the ability to accept Commuter Check cards, I found out that one of my two cards was “malfunctioning.” According to the clerk, this happens all the time. I happily paid the remainder of my monthly pass with my own credit card in order to end the whole painful ordeal! I returned the bad card to my company and recommended that they no longer issue Commuter Check cards (see Warning). In any case, I won’t be using them again.

I’m very excited to share that Pulse has announced it’s series A funding round! All of us are still fired up about last week’s Apple Design Award at WWDC and our recent 4 million user milestone, not to mention that today is our co-founder Ankit’s birthday. Thanks to the team for their tireless work and to everyone who has helped us get here!

Check out some of today’s press:

Pulse Blog – Announcing Our Series A Financing
TechCrunch – 4 Million Users Strong And Apple Design Award In Hand, Pulse Grabs $9 Million Series A
WSJ – Pulse Taps $9M To Win Battle For Mobile-News Consumers
Forbes – News Reader Pulse Raises $9 Million
Mashable – Pulse Passes 4 Million Users, Raises $9 Million for Visual News Reader

Upon hearing about my sleep deprived state, my friend sent me a link to the top 5 regrets people make on their deathbed along with the comment “you might need this.”  I appreciated the link and enjoyed the reminder to live life to the fullest, especially with regards to keeping in touch with friends and loved ones. I also realized that my friend didn’t understand that for me the long hours I put in are all about fulfilling my dreams of creating new technology and impacting the world in a positive way. According the article, not chasing after dreams is people’s #1 regret.

Of course there is an opportunity cost to time spent on any endeavor and this inevitably contributes to spending less time with friends and loved ones (regret #4). I believe maintaining a healthy balance between the two is critical. Simply “working less” (regret #2) would not make me happier. Chasing after dreams is an essential part of my life. The feeling of fulfillment I get from doing so makes me a much happier / more content person, and this in turn positively affects my relationships.

However, sometimes I do get caught up in chasing my dreams and forget to make time for friends and family. Just like realizing dreams, successful relationships are built on quality time spent together. I always appreciate being reminded to dedicate more time to this essential part of life, as I was today. I’d love to hear your thoughts or personal experiences on achieving the right balance.

Disclaimer: This post was written in a sleep-deprived state.

After reading some suggested approaches, I spent more time than I would have liked fighting with Git to actually make it happen. In the hopes of helping someone else avoid the same trouble, here’s the solution that ended up working best. The solution is primarily based on ebneter’s excellent question on Stack Overflow.

Another solution is Linus Torvald’s “The coolest merge, EVER!“ Unfortunately, his approach seems to require more manual fiddling than I would like and results in a repository with two roots. I don’t completely understand the implications of this, so I opted for something more like a standard merge.

Goal:

• Move directory 1 from Git repository A to Git repository B.

Constraints:

• Git repository A contains other directories that we don’t want to move.
• We’d like to perserve the Git commit history for the directory we are moving.

Get files ready for the move:

Make a copy of repository A so you can mess with it without worrying about mistakes too much.  It’s also a good idea to delete the link to the original repository to avoid accidentally making any remote changes (line 3).  Line 4 is the critical step here.  It goes through your history and files, removing anything that is not in directory 1.  The result is the contents of directory 1 spewed out into to the base of repository A.  You probably want to import these files into repository B within a directory, so move them into one now (lines 5/6). Commit your changes and we’re ready to merge these files into the new repository.

git clone <git repository A url>
cd <git repository A directory>
git remote rm origin
git filter-branch --subdirectory-filter <directory 1> -- --all
mkdir <directory 1>
mv * <directory 1>
git add .
git commit

Merge files into new repository:

Make a copy of repository B if you don’t have one already.  On line 3, you’ll create a remote connection to repository A as a branch in repository B.  Then simply pull from this branch (containing only the directory you want to move) into repository B.  The pull copies both files and history.  Note: You can use a merge instead of a pull, but pull worked better for me. Finally, you probably want to clean up a bit by removing the remote connection to repository A. Commit and you’re all set.

git clone <git repository B url>
cd <git repository B directory>
git remote add repo-A-branch <git repository A directory>
git pull repo-A-branch master
git remote rm repo-A-branch

Update: Removed final commit thanks to Von’s comment.

A few months ago I mentioned that I left the government/research world (Sandia Labs) and joined an exciting new startup.   I’d like to share a bit more about my experience so far and announce that we are hiring!

Those who have worked at a large company and then moved to startup can probably relate to my experience.  First, without a doubt, the most motivating and fun part about working at Pulse is seeing the impact of my work. And I don’t mean just having someone say “Good Job” or receiving a strong performance review, I mean seeing thousands of people USE the results of your work and submit feedback about how it benefitted their lives.  At Pulse, this experience is magnified by the fact that we release new product features every two weeks, and not ever quarter, or every year!

In addition to really seeing the impact of your work at Pulse, another benefit I never knew I was missing in the corporate world is the feeling of being a part of a tight-nit team where everyone is giving 110% towards reaching the same goal.  This is something that is rarely found outside the startup world and should not be underestimated in its ability to improve productivity and genuine enjoyment of work. Imagine you and your coworkers are always committed to the same goals and never distracted by office politics or personal agendas.  How often does that happen in your current job?

Finally, at Pulse both the impact of your work and great team dynamics are highly leveraged.  In the corporate world great project outcomes or solid team building commonly result in a plus mark on the rating form and a promotion in three to five years.  How motivating is this for the most capable employees?  The impact of their work, which in the computer science world can easily be many times that of other employees, is muffled by ridged performance review and compensation structures.  In a startup, this muffling effect is removed.  Everyone accepts slightly lower fixed compensation up front in exchange for the chance to turn hard work into a big reward down the line.  This reward is directly tied to the impact your work has and how well the team works together.  Now everyone is properly incentivized to give their all, and giving your all for something you believe in is inherently rewarding!

Check out the official Pulse Blog and our hiring post there.  Here are some of the postions we are currently looking to fill.

OPEN POSITIONS:

iOS developer: Our iPad/iPhone platform continues to be our most active platform. We’re looking for a developer to join our iOS team, which currently consists of Ankit (our co-founder) and Dr. Tyler (yes, he has a PhD from NYU).

• We don’t care much about your college, your major or your GPA.
• We do care about your experience with the iOS platform. Ideally, you have already developed and launched an app in the store. You prototype features rapidly and iterate on the design even more rapidly – while writing clean code.

Web developer: Pulse currently does not have much of a web presence. We hacked a Wordpress theme to build our current website. We use Posterous for our blog. For the rest, we depend on Get Satisfaction, Facebook, Twitter and the likes. You will be leading our charge on the web!

• We don’t care much about your college, your major or your GPA.
• We do care about your experience with web. Ideally, you are experienced in writing HTML/CSS/Javascript (GWT is a bonus!). You’re also familiar with either Django, PHP/Python, AppEngine, RoR or other frameworks. You love making beautiful websites or web applications. Huge bonus if you’ve already made some.

Drop us a line at jobs@alphonsolabs.com or feel free to ping me directly at gb [at] alphonsolabs.com

What about the big data analytics and informatics that made Google famous? Does AppEngine help us there as well? The answer is yes; although with some serious limitations.

AppEngine MapReduce

Recently Mike Aizatsky and others at Google started working on a great add-on to AppEngine called MapReduce. This is, of course, conceptually the same MapReduce originally published by Jeffrey Dean and Sanjay Ghemawat here and the same MapReduce upon which Hadoop is based.

Unfortunately, for various reasons, Google decided not to make their internal MapReduce infrastructure available. Instead, they have decided to develop an entirely new system within the sandboxed world of AppEngine. This allows them to provide some additional batch processing support, while protecting their infrastructure.

For AppEngine developers, this means that a limited set of analytics functionality is currently available out-of-the-box. The new functionality amounts to support for “map-only” jobs at medium scale (64 shards). Reduce functionality, which is critical for most analytics, is not yet available.  If your application requires reduce-like aggregates, your options include: waiting for Google to release more features, switching to Amazon EMR or EC2, or working around Google’s limitations.

Assuming you chose option number three, there are several ways to implement aggregates on top of AppEngine’s map jobs.   As an example, lets look at a simple word count.

Built-in Counters

To count how often a few words (up to about a thousand or so) appear in a large set of documents, we can use the platform’s built in counters.  The code looks something like this or this:

from mapreduce import operation as op

def process(entity):
    word1 = entity.content[0]
    word2 = entity.content[1]
    yield op.counters.Increment(word1)
    yield op.counters.Increment(word2)

Unfortunately,  for large aggregates this won’t work.  Assume now that you want to count all of the words in the documents.  There will be too many counters to use the built-in facility.  Since AppEngine does not support reduce operations, the usual approach of allowing a reducer to sum up counts from each mapper is also unavailable.  What remains is to implement an efficient, scalable counter.

Sharded Counters

Because of the parallel nature of many mappers counting at the same time, one resonable approach is to implement sharded counters.  The goal is to avoid database contention by splitting a single counter into several shards.  These shards can then be incremented independently and summed up to get a total counter value.  This technique can be quite effective, but still requires a database read and write for every counter increment.  In practice I found that this severely limited the throughput of my MapReduce jobs on AppEngine.

In-Memory Counters

To avoid blocking on database reads and writes, a logical approach is to consider using the low latency memcache layer AppEngine provides to implement completely in-memory counters.  This works great for short jobs with a small to medium number of counters.  Unfortunately, when running a long job with many counters, it is very likely that some (sometimes many) counters will be evicted from the cache before the job completes.  Since AppEngine only allows cache eviction policy hints and shares one big cache among all apps, there is no way to prevent this.

Write-Behind Counters

For long running jobs with many counters, we need a combination of low latency (in memory) counters and durability to survive cache evictions.  One elegant solution is to use AppEngine’s task queues to write counter values to the datastore periodically.  This allows counter increments to continue at high throughput without blocking on datastore interactions. Instead, updates to the datastore are batched and written back less frequently.  This works well because the memcache is unlikely to evict recently incremented counters before the background task has a chance to store the counter’s value.

Note: With this approach there is a small chance that a counter value will be lost.   However, for many applications this is acceptable and well worth the increased mapper throughput.

Pulse is currently the #1 news reader on the iPad, iPhone, Andriod app stores.  I’ll be leading the development of our backend data platform and working with a great team.

As we start to build out Pulse’s backend, I’ll be continuing to experiment with Google App engine.  Stay tuned for more posts in that regard.

Thoughts?  Please share your comments below.