How The $1million Netflix Prize Spurred a Radical Innovation, Accidentally

Amazon's 'People who Bought... Also Bought' feature has become something of a modern totem. If you don't have recommendation you are not truly digital.  You might sell online but you don't cross-sell and upsell and you don't get your customers to sell.  In fact you don't appear very social and nor are you in tune with big data.  Recommendation is a window onto the digital soul - yours and your customers, how we appeal to people and how people influence each other. Selloscope, the new SaaS, data crunching recommendation engine out of Dallas, is an aptly-named guide to this religion and CEO Jeb Stone is both fun and a source of insight on what we do when we buy. Our paths crossed on the subject of Netflix and the Netflix Prize, a landmark in open innovation.

In the mid 2000s Netflix launched a grand prize contest - improve the Netflix movie recommendation engine by 10% and win $1 million. Thousands of teams pitched in with their ideas and the award went in 2009 to BellKor’s Pragmatic Chaos.

Jeb Stone noticed that the ideas in the Prize tournament were driving along a similar path. In fact constrained by the competition's rules those thousands of teams were in danger of proposing a solution that would not be optimal, in part because they were driving after error reduction in the existing system rather than developing a new understanding of how people share information about products, likes, services, content. A winner might improve Netflix's recommendation engine by 10% but what if it could be improved by 100%, 200% or more. Why close the door to exponential improvement?

The result of that thinking is Selloscope - Stone's start-up recommendation engine in the Cloud.

The web takes away the cost of sharing information about what we consume; it brings out the gossip mongers in us - on an historic scale; we embrace sharing like we've never done before. And we can do more than we every dreamed possible.

Stone's observation was that consumption doesn't stop with the act of watching a movie or dabbing on the perfume or taking the shot with a new camera. It is at least as much about the post experience story, the tall tales we tell each other and the passions that those experiences inspire in us.

Sociological experiments show that whatever group you belong to the people around you influence your purchases. When assigned to random groups to select music with people you do not know, for example, you will collectively create a shared favorites list that is likely to be quite different from the lists of all other groups that might form for a similar task. In other words social pressure is an essential ingredient in shaping what we like or favorite and what gets to be the most popular is a consequence of who you are with. Somewhere along the line we need to get under that influence system to understand what people really do enjoy.

Selloscope tries to deal with it by using co-purchasing patterns - Stone uses the absence of things in our shopping cart to tell us what we are missing in our buying process that is not missing in others. Recommendation engine technology though is a mystery to me - but setting innovation challenges is not.

Selloscope joins a small army of start-ups in the recommendation space, maybe a few who were also inspired by the Netflix Prize but not the way we might have expected. An unintended consequence of the Netflix challenge though is more knowledge on how we think and act together, as well as more recommendation engines. But it has also spawned an important lesson in designing challenges and open innovation projects. We should never set the rules in stone.

My Top Three Links:

A round up of recommendation engine links and thinking.

Answers to recommendation engine questions on Quora from Emmanual Marchal, Director of LikeCube.

Jeb Stone's piece on why to do it his way.