Category Archives: Uncategorized

Notes from OpenStack Summit: Tapjoy’s Wes Jossey

Last week, I attended the OpenStack Summit in Paris. I wanted to capture key notes I had from the sessions I found most interesting and share them. I’ll also provide a general recap on my own observations. I’d encourage others to write-up their observations and notes. I’d be happy to share and link to interesting write-ups, as they might be interesting to both of the readers of this blog. ☺ I’ve read Mark Potts’ great recap here, as well as this post by Forrester’s James Staten.

For this post, I am recapping Wes Jossey’s session (video) on Tapjoy-1 on OpenStack (here’s a link to the deck as well). Tapjoy is a leading mobile advertising network company. Tapjoy-1, as I understand it, refers to a private cloud, running a core Tapjoy workload (i.e., the bulk of their mobile ad network). Though it had run entirely on Amazon Web Services (AWS) and Tapjoy had been an early AWS adopter, Tapjoy now runs both AWS and Tapjoy-1, its OpenStack based private cloud, provided by Metacloud.

Tapjoy is finding business value success running Tapjoy-1 on OpenStack, as for the same cost they were paying AWS, Tapjoy has gained much higher output on OpenStack in a private cloud (see chart below).

Here’s what this chart says. Assume same amount of spend over 3 years. The blue bars signify the amount of capacity provided in terms of nodes, IOPS, storage, etc on Amazon Web Services. The red bars signify the same amount using OpenStack. The deltas are significant across all 5 major vectors. Eyeballing the chart, I estimated that Tapjoy achieved a doubling of cores, 4x increase in RAM, 10x increase in IOPs, 50% more disk, and 10x increase in nodes. This is the “money” slide, so to speak, in that it makes OpenStack seem like a no brainer, certainly for Tapjoy.

Wes indicated that he and his team penciled out 3 years of spend on their private cloud, including hardware, software, services, floor space, power, etc. He also stated that Tapjoy had experienced better uptime with their OpenStack private cloud implementation than they had when they’d run the app on AWS or on Bare Metal.

The Tapjoy experience and case study is an important bellwether for the OpenStack movement. Despite much excitement and momentum around OpenStack, as a community, our project is still early in full production design wins with concrete business value evidence. Tapjoy’s experience and the evidence presented are a solid example. I wanted to share this and provide some of my own perspectives.

What is Tapjoy-1

Tapjoy-1 is an internal private cloud, built on OpenStack that all Tapjoy devs have access to. It is comprised of 348 “Data” All Purpose Nodes (see slide below). According to my notes, Tapjoy-1 has 12 management nodes. As I understood it delivers a key production workload for Tapjoy, namely delivering mobile ads.

Tapjoy’s Observations / Lessons Learned

Wes shared several observations and lessons learned in his talk, which I’m capturing in no particular order.

Lesson 1: Vendor selection matters

Wes was a happy reference for Metacloud, and he described how they helped validate the design, and then they did the OpenStack deployment and provisioned the network. Quick commercial plug: HP Helion is now open for business, and we can offer these capabilities as well! ☺

Wes was also a positive reference for Equinix.

While he didn’t bash Quanta directly, he did say that delays in procuring hardware blew out both his timelines and his hardware contingency budget.Quick commercial plug #2: HP is a leading server and hardware vendor, and we can *definitely* help here! ☺

Lesson 2: There’s nothing agile about writing a big check up front. At same time, there is something smart about planning through all the money.

Wes’s point here was one he covered only briefly, but it has stuck with me. Tapjoy made the bet to move to an OpenStack based private cloud, specifically choosing to go from from Opex (pay as you go) to Capex (invest up front). With this, he was pointing out that they had to do a lot more planning up front. While it was less agile and real-time than just adding or shrinking capacity as they went along, I think his point was that smart planning, plus the compelling economics made this a stark and valuable business case. I expect we will see more examples of this in the market and the community.

Agile is a very popular and often appropriate methodology, and in many cases it will work. But in this case, and I suspect in many others, particularly where the app and workload are well understood, planning up front and shifting to Capex was a no brainer, business wise. Quick commercial plug #3: HP has financing options available across hardware and software, so if you want to build a private cloud, we can help you do so either in Capex (pay up front) or Opex (pay-as you-go).

Lesson 3: Challenges

As is common with many new IT initiatives, selling internally a change can be a challenge, and Wes hit on this theme. There had been no previous success story with OpenStack internally, so Wes and his team encountered a lot of skepticism. In addition, Wes cited frustration with turn around time (again principally on hardware delivery), as well as on the need to plan out all the different steps (Wes said, “you’ll need a Gantt chart”).

Lesson 4: Design Principles for Cloud

A few key points here. First off, was this message: plan for failure.

Wes talked about Tapjoy’s preference of thinking and trusting of application more than disk replication. In other words, they think of replication occurring at the app layer. They don’t use CEPH, as they want to think of things as ephemeral, and don’t want to be required to re-mount disks. (He also mentioned that Tapjoy doesn’t use AWS’s EBS for similar reasons, namely it doesn’t seem very reliable.)

He also discussed the concept of service boundaries. He described that there were 20ish services that were connecting, and his point was that you just have such a broad mesh of potential failure points, that you have to just avoid an assumption that the other side of the service is healthy. In the architecture of the app, he had the concept of the Circuit Breaker, where if a service were down, there was an alternative flow that the architecture followed. This was Tapjoy’s architectural design to implement the reality that they couldn’t assume that all 20 services were always going to work, it is kind of a real time backup plan. He also talked about having hardware and software contingencies, backup links, and temporary caches. Finally, he advocated testing failure in production.


Basically what I’ve described above you can see watching the video of Wes’s talk that’s linked to at the top of this post. Here is my perspective.

Perspective 1: I’d expect that we will see more examples of compelling business evidence like this from other early adopters.

This was my third OpenStack Summit, starting at Hong Kong a year ago. Wes’s talk was interesting to me, as it was really the first one I’ve sat through with OpenStack running in production, at scale, on a central workload for the business. (Others have talked more at a lab or dev/test level, though I may have missed sessions at some point.)

Wes and his team at Tapjoy are early adopters for sure. But the business results they are realizing are pretty breakthrough. While an early example, it is an important one. The results are providing so much daylight between the status quo of just using AWS or other proprietary offerings, that I expect more companies will have to start looking at OpenStack on a private cloud as an option.

Perspective 2: We’ll see more discussion on agile on demand v planned cloud deployments

Mobile adtech startups like Tapjoy exist on the basis of being disruptive and moving quickly. Speed and fast iteration are hallmarks of tech startups. And this speed and agility is a Good Thing, not just for startups, for anyone.

With this ‘move fast and break things’ ethos, the attraction of public cloud, pay as you go pricing is compelling and natural. You can spin up resources as customers demand those resources.

At the same time, savvy IT shops will seek to optimize cost and value. This should be especially so for tech startups, who generally have to be very aggressive at hacking the most cash efficient options available to them, as the cost of capital for venture funded startups is so high.

So while the image of hacker tech startups just choosing to go Opex and AWS and calling it a day is fashionable, the Tapjoy example provides an interesting contrast. Leading VCs have warned recently that a reckoning is coming in terms of startups and that burn rates are getting out of control. With that as context, you’d have to forecast that more startups will evaluate whether they can reasonably achieve Tapjoy-like economics going private cloud/Capex versus Opex.

For more mature enterprises, who are seeking to drive more agile methodologies more broadly, it will be interesting to watch how this discussion evolves.

Net/ I’d expect that we see more companies coming forward in the next 12 months with similar types of discussions to the Tapjoy example.

Perspective 3: More discussion on building and architecting cloud native apps

Wes had some very useful perspective on building a cloud native app with the assumptions that things wouldn’t work. In particular, and I heard this in other sessions too, Tapjoy had architected its app such that replication was occurring at the app layer, that this obviated the need to replicate storage and remount disks. For those farther down the path of going cloud native, this is known. This still seems an emerging practice, however, as I talk to customers often who are asking about data and disk replication. I think this focus on taking this to the app layers will need continued focus and push.

Really enjoyed Wes’s session, and I’d like to thank him and the Tapjoy team for putting together such a useful case study.

Anyone else got thoughts for this session or others at the OpenStack Summit?


I’ve had a difficult time really gearing up today to scour the tech industry landscape for a perspective to share or anything interesting to write about.  Newtown is too top of mind.  Too sad.  Too awful.  I’ve been reading a lot of perspective on the tragedy, and here are some thoughts.

Of all the opinion I’ve read, I thought Buzz Bissinger‘s piece in the Daily Beast was the most stark, the one that I’m going to save and keep nearby.  It’s classic Bissinger: pulling no punches, Bissinger isn’t for someone who wants a pat on the head and told there will be a silver lining.  No, he takes the culture to the woodshed in the article that he condlues:  His concludes the piece, titled, “Sorry But Don’t Expect Any CHange After Newtown,” this way:

We cannot let go of the violence.  And as long as we cannot, these tragedies will continue, seemingly inexplicable—but not really, given our history. There is an answer. We could ban handguns as the United Kingdom did after a mass murder at a school in 1997. We could weigh the societal priorities of gun ownership, and psychotic killers getting access to semiautomatic rifles with the most potent ammunition available, versus not ever allowing an innocent child to die again for the sin of going to school and getting ready for the holidays.


But we won’t…  Guns in this country still pump people up.People still like the concept of taking the law into their own hands because the law is a toothless pussy, the giddy thrill of an intruder coming into their house, assuming he is an intruder because he may not be, and blowing his head off with the bonus extra of brain-matter spattering on the wall.

I fear that he could be right, and that would be too bad.

There have been other, more productive, threads of thought that I’ve been following.  Raising more awareness on the issues of mental health in children and young people.  Discussing gun control and the need to do more, much more, here.

And these are both vitally important discussions.  I will watch and listen to these debates closely and try to be useful in whatever way I can.  And I will keep Bissinger’s piece nearby  hopefully as a prod to lean forward to try and push more and demand more of our government and all of us, to not let our violent past dull us into the continuing sameness of accepting the unacceptable.


Enhanced by Zemanta

Happy Thanksgiving

Happy thanksgiving all.  A favorite American holiday, Thanksgiving is a  an opportunity to watch football, eat turkey and reflect on our circumstances, what we’re thankful for.

Every year, I look forward to re-reading The Desolate Wilderness, published every day before Thanksgiving by the Wall Street Journal.  If you’ve never read it, I’d encourage you to read it.  It’s a great account of the first settlers/pilgrims and their journey from Holland to the New World, and the crazy foreignness of what they experience here.

This section in particular always impacts me:

Being now passed the vast ocean, and a sea of troubles before them in expectations, they had now no friends to welcome them, no inns to entertain or refresh them, no houses, or much less towns, to repair unto to seek for succour; and for the season it was winter, and they that know the winters of the country know them to be sharp and violent, subject to cruel and fierce storms, dangerous to travel to known places, much more to search unknown coasts.

Besides, what could they see but a hideous and desolate wilderness, full of wilde beasts and wilde men? and what multitudes of them there were, they then knew not: for which way soever they turned their eyes (save upward to Heaven) they could have but little solace or content in respect of any outward object; for summer being ended, all things stand in appearance with a weatherbeaten face, and the whole country, full of woods and thickets, represented a wild and savage hew.

Whenever I read this, I find it bolsters me.  Be not afraid is the message I take.  No matter what challenges one finds in front of him or her, they likely pale in comparison to this account.  Showing up on the banks of New England, with literally nothing established.   Truly, the ultimate startup.
Be not afraid.  Have a Happy Thanksgiving!
Enhanced by Zemanta

Quick thoughts on last week’s Presidential Debate

Last week, I was home taking care of the kids, and had TiVo‘d the Presidential Debate, figuring I’d watch it after homework was done and kids were asleep.  Like most, I thought Obama had this thing sewed up, and I expected the debate to be a low risk affair in which Obama rose above the fray and continued his glide path to a re-election.

As I was getting kids into PJs and checking homework, I did glance at Twitter as the debate was going on.  Romney was clearly winning, based purely on the Twittersphere. I later did get to watch the debate.  3 quick observations:

This wasn’t a debate, it was a performance review meeting.  I think this was one of the two most damaging things about the night for Obama.  Listening to Romney’s tone and approach, along with Obama’s lack of fight and rebuttal, left me with the impression that Romney was a manager telling his employee that his performance wasn’t making the grade and he needed to be fired.  Romney’s tone, approach, and manner are very similar to how a professional manager goes about telling someone that they aren’t making the grade and need to be fired.  It’s specific, thorough, direct.  Obama basically assented on most of these critiques, leaving the impression that he basically agreed.

Mute the audio, and forget which of the two was President, then ask yourself which one is President.  This is the second big wound the Obama campaign took, in my view.  If you buy into the Malcolm Gladwell concept of Blink, then you believe that people make snap decisions based on very quick impressions of people.  This can be applied to hiring, to whether you like someone, etc.  I think if  you were able to set aside whichever candidate you favored, you turned off the audio, and you just watch both candidates for basically any 30 second segment, you’d probably conclude that Romney was the more Presidential, whatever that means.  This can’t be verified, but I do think that this is a big impact.  For the first time, people saw a picture that was so different and so jarring.

Complaining about a moderator is like an NFL coach blaming the refs for a loss, it’s a waste of time and it just reinforces that you’re losing.  During the debate, the big surprise was Obama being so flat and unprepared to debate.  Post debate, the big surprise, to me at least, was how much criticism was heaped on Jim Lehrer, the moderator.  To me, this is like complaining about the refs in football.  It sets the wrong tone.  I recommend Pittsburgh Steelers Head Coach Mike Tomlin’s approach.  Whenever we lose, he says basically, “We lost.  We accept that.  We are not happy about it.  It is what it is and we accept that.  We have a lot of work to do with urgency.”  And that’s that.  He moves on.  That’s the recommended path.  Don’t spin when it makes you sound like a sore loser.

A final conclusion… As a free markteer, I generally favor competition.  We now have some in this race.

Enhanced by Zemanta


Graffiti of Edmund Hillary on St Benedicts Str...

After several weeks on the road, I’m back in my beloved California.  Lots to talk about from the travels–notably, wow is New York a hotbed of strong startup activity!  But for now, just happy to be back on the West Coast.

Always reminds m

e of this great quote, which I took from Sir Edmund Hillary‘s obituary a fwe years back:

I discovered that even the mediocre can have adventures and even the fearful can achieve. I had the world beneath my clumsy boots and saw the red sun slip over the horizon after the dark Antarctic winter. But for me the most rewarding moments have not always been the great moments, for what can surpass a tear on your departure, joy on your return, and a trusting hand in yours?

Enhanced by Zemanta