Want to build a service that keeps adding new features and fixing bugs and generally getting better? The biggest problem might be your most experienced people, who know all the ways things can go wrong.
Whether you call it devops or continuous deployment or just cloud services, the way Google and Netflix and Amazon keep experimenting and changing and fixing and updating their systems sounds almost too good to be true. "People say 'it's not possible – and I know because I've done this all my life'," admits Bing architect Craig Miller.
Proof from rivals
Ironically, it was Google that pushed Bing to learn to update so fast in spite of those attitudes. "We wanted to do it daily but at that point we shipped monthly. Everyone said 'you can't do that'. But we had the benefit of competitors saying they were doing some of this, so we had at least some proof points that say it's possible."
Bing had the advantage that it had already built a system where it could run experiments with the service and try things out. "We already had the experimentation infrastructure – we've got a very sophisticated a/b testing system we can run. We run thousands of experiments a day." In fact, Miller says, "we're experimenting constantly on you."
Some of those are big changes – Miller was the person who first used a video loop for the Bing image of the day. Others are much smaller. "On the results page we might make one 2 pixel change and measure that. With the level of experimentation we have, if I change a single pixel, you would see an effect. If I reduce engagement with that change, I'll see that signal," he explains. There are some things that always make searchers less happy, the worst one being pushing things down the page so people have to scroll. "Scrolling is verboten," he notes.
Not all of the changes will work out, and Miller is fine with that. "Ideas are not all good. When I have the velocity, I can try it out for a couple of days and discover that didn't work and it was a stupid idea."
Zero blame policy
For one thing, the same system that lets Bing try new things several times a day lets them change things back quickly. And whether it's a bug or a bad idea, there isn't going to be any finger pointing. "Zero blame," emphasises Miller. "It has to be a learning culture, a culture that says 'we're going to learn from this'."
In fact, he's happy if most of the ideas don't work out. "How often do we succeed? How often do we ship a feature we have put in front of you in a test? 10% of the time." Think that sounds low? "Every company in the world would be ecstatic if they knew 10% of everything they're going to do will be successful," Miller claims.
The real advantage of all those experiments and updates is how quickly Bing can make changes. Miller notes: "Ideas are cheap, but if it's expensive to implement them then you don't do it. You have to have the ideas, you have to have the quality and you have to have the ability to test them and experiment. A lot of things come through and if they fail that's okay because you've got a culture of learning fast.
"If we have a lot of ideas and deliver them faster than our competition, we will win. At the end of the day, that's all it is. If we can deliver faster than the competition, we're all good."
