In his Mobile Patterns for Experiments presentation at Google Conversions 2017 in Dublin, Ireland, Craig Sullivan walked through a set of optimization testing best practices for mobile experiences and why they are key to business. Here's my notes from his talk:
- What has manufacturing taught us? Order doesn't come by itself. The orchestration of people and stuff needs to be managed, this is what has been driving efficiency in manufacturing. This efficiency is critical to survival and used to: improve quality; remove defects & waste; increase capabilities; manage resources; plan resources; etc.
- What has retail taught us? Thousands of tiny little details must work for you to have a good experience. There are many retail optimization components: window, greeting, lights, heating, displays, etc.
- What are the atomic parts of cross-device optimization? navigation, search, login, registration, forgot password, etc. These can and should be optimzied to ensure return on investment and survival.
- We don't learn enough from manufacturing & retail. No one has a meeting where stuff is removed from a Website. We just keep adding stuff. Many small things in digital experiences are broken: touch target sizes, overlays, errors, layout, etc.
- How can you increase conversion? persuasion, motivation, usability, and broken stuff (start from the end and move backward). An experience is only as good as the crappiest part. A/B testing will only help you polish a turd, if that what you have to being with.
- Product defects are very costly. Find bugs, fix them. It's a clear and fast return on investment.
- We change stuff without evidence or observation. We build stuff nobody wants or needs. Product changes are not measured or tested. We build what internal people want, not customers. Ego is the enemy of all good work. A lot of Internet design is infused and driven by Ego.
Testing & Optimization Power-ups
- Don't test unless you have enouh traffic for it to matter. Grow first.
- Don’t copy stuff blindly: everything is unproven until tested against your specific audience/product.
- Prioritize your test ideas. Score all your testing ideas, then decide what to focus on.
- Get a hypothesis: Because we saw [data/feedback], we expect that [change] will cause impact, that we can measure using [metric/data].
- Record the design people see, tests can look different across all devices. Build, borrow, or use a third party service: how does your experience look on different devices/mobile?
- Audit your Google Analytics to make sure it is set-up properly.
- Get a writer. Copy writing is a real job, you’ll make more money than you spend.
- Look at the big picture in addition to the details. Both these spaces help you solve problems.
- Stop wasting money on things that don’t work; or aren’t actually needed.
- What is very simple and converts well… is really hard to do well.