Conversions: Building a Testing Culture

by March 24, 2017

In his Building a Testing Culture presentation at Google Conversions 2017 in Dublin, Ireland, Stuart Frisby discussed the culture, process, and team structures needed/used at Booking.com to support thousands of optimization tests. Here's my notes from his talk:

  • Booking.com's use of testing results in 2x-3x industry average conversion rates.
  • If it can be tested, test it. If it can’t be tested, don’t do it. No platform goes untested.
  • Teams are made for testing a hypothesis. They're assembled based on what they need to vet a hypothesis.
  • Everyone has access to as much data as possible, which they get at Booking.com. Access to data allows more hypothesizes to emerge, which can then be tested.
  • There is a difference between ideas & hypothesis. Ideas aren't useful. Hypotheses are grounded in the reality of your business and need to be proven or refuted.
  • Test small: too many changes at once makes it difficult to know what had impact. Most tests fail, so need to be able to move quickly and isolate insights.
  • 9/10 of tests don’t have positive impact and you don’t know why. This causes a lot frustration.
  • You have to keep spending money on research; this serves as an input into the A/B testing process. It creates more hypotheses to test.
  • Booking.com has more than a 1,000 tests running at any given time.
  • There’s a temptation to limit tests to 5% vs. 100%. You’ll need to go to 100% at some point, so might as well learn as much as possible as fast as possible, and not limit test sizes.
  • A/B testing is predicated on trusting the tools you have. Without trust in tools, you can't have a culture based on testing.
  • 75 product teams at Booking.com each made up of complete teams of designers, engineers, product owners, writers, etc. They are able to tightly integrate with horizontal support teams for each of these domains to create community, training, and more.
  • What kind of skills do you need in an A/B testing organization? entrepreneurialism, arability, humility, curiosity, statistics, analysis
  • Throw out your roadmap, and trust in the process and your people. Go where the tests take you. That may end up in a different place than you envisioned but it is where your customers are taking you.
  • You need to ensure people know what is the right thing for your business. Invest in training. You can't have a culture of trusting your people, if you don't train them in how your business works.
  • As a leader in a testing organization... are you willing to be wrong? Are you willing to be told you are wrong? Repeatedly?
  • Scale of impact of individual tests is quite small but when applied to a large scale business, it matters.