At the Web App Masters Tour in Philadelphia, Julie Zhuo talked about Facebook’s approach to Data-Informed vs. Data-Driven Design Decisions. Here’s my notes from her talk:
Facebook believes in small teams that can act like a start-up. The team on the ground needs be the most knowledgeable about what data to use on their project. Each product usually has a single designer assigned to it. There are only 20 designers and 5 researchers in the company.
Using Quantitative Data
- Facebook uses quantitative data to inform decisions, help understand how people are using the product, and how can they optimize key flows further.
- The company kind of got into the data game late. From 2004 on, the company just kept adding feature after feature. They never took a step back and looked to see what was successful. Some features were doing a lot better than others.
- It wasn’t until 2009 that Facebook went back and updated their Photos service. The last they touched it was in 2006. Photo uploading was hard to do. Facebook used a Java uploader that was built by an outside firm.
- When the team redesigned the photo uploader, they built a solution that required the ability to install a plugin one time and from then on photos could be uploaded. This implementation took about 4 months to build.
- The new photo uploader was rolled out to small % of users. When they loked at the data, only 34% of users that started were successfully able to upload photos. It was the install plug-in step that was tripping people up. Half of people dropped of at the install step.
- The team decided they needed to go back to what people were familiar with. They had to throw away months of work and instead stick with what people knew. The second iteration of photo upload made use of native OS controls.
- In the second redesign, of all the people who hit the upload photo button, 87% made it past the create album form, 57% selected photos, 52% make it to upload step, but only 45% of people actually uploaded photos.
- The more standard design increased uploads from 34% to 45% but the team thought they could do better.
- They found 85% of people were only selecting one photo at a time -probably as a result of the native dialog. People needed to know how to use the OS control to select multiple images.
- The Facebook team designed an interstitial to educate users about how to select multiple photos in the OS dialog. Though they usually avoided interstitials, it was important enough to try.
- With the interstitial tip in place, the number of users uploading only 1 photo dropped 40%. Photos uploaded per attempt increased from 3 to 11.
- Another example of using quantitative data to inform design was the “composer” –where people publish status, links, etc. For the composer, the team A/B tested explicit actions for: photos, questions, links and status updates. In this design, Status updates decreased 1%, photo uploading went up 1.5%. But there are way more status updates than photos. So the offset didn’t pay off.
- The second idea they tried was to show a user’s last status update at the top. The idea was if you see an old update you might want to update it. This approach didn’t have any effect on updates.
- Another option included links and an exposed status box. This performed better but only marginally.
- After spending a lot of time tweaking the composer and not getting a lot of results, the team finally stopped working on the problem and moved on to something where more gains could be realized.
Using Qualitative Data
- Quantitative data points to insights. Qualitative data can be used to optimize and sanity check flows.
- Facebook’s first user researcher joined in 2007. In the early days, they tested people going through flow. But because employees at Facebook use the product so much, they spot issues before usability sessions and fix them. As a result this type of qualitative research is not as valuable to them.
- Instead the user research team focused on bigger picture issues.
- Recently studied identity and what people feel comfortable sharing through ethnographic studies. Also studied local businesses to tailor pages product to these smaller businesses.
- The end results of qualitative research are insights on how people feel and what they think. This funnels into higher-level strategy.
Evaluating Design with Metrics
- Some projects at Facbeook use data to judge their success. Not all projects are judged this way but the growth team is an example of a completely data driven team. The growth team was created in 2007 and was the first example of data driven design at Facebook.
- In 2007, Facebook opened up to the world (beyond colleges). A team was formed to tackle growth. A social network is valuable if people you know are on it. Otherwise, you don’t know what it’s for.
- Growth team looked at registration, metrics associated with people that were coming back, etc.
- A simple visual design change on registration page resulted in 9M more people per year joining. 3% increase. For growth this is a big deal. Deactivation page. Reduced deactivations by 7% (a million more users a year).
- A few months ago, Facebook created a team called engagement –modeled after the growth team. Thought it would operate the same way. First tried quantifying success by looking at number of read/writes. One of the early ideas was comment liking. Saw an increase of writes in the system 7% up when this feature was rolled out.
- Though writes went up 7% it was the same people creating writes. 85% of reads/writes of Facebooks are generated by 20% of users. Same people were doing the activity. Had to go back and reset metrics for the engagement team.
- Current metric for engagement at Facebook: L6/7 number of users that come back to Facebook 6 out of 7 days a week.
Making Decisions
- Facebook is wary of being too data driven. Data alone might look good but could have a bad impact on brand.
- It’s difficult for metrics alone to represent what you are trying to build with a company. Decisions at Facebook have to balanced between: qualitative data, quantitative data, strategy, user (individual interests), and network interests, competition, regulatory bodes, and business interests.
- You need to think about all these things as you design.
- Quantitative data helps understand how people use products, how to optimize flows, and if products are successful.
- Qualitative data helps understand user intents and feelings in order to focus strategy.
- Data is one small piece of the pie and lots more comes into the decision-making process. True innovation does not come from being lead by numbers.
- When launched, the Facebook newsfeed had an overwhelmingly negative response from users. Metrics plummeted the day after the launch as people were not coming to the site. The team made some modifications to give users more control but never got rid of the idea even though metrics took a hit at first. Now the newsfeed is hugely successful.
- Facebook learned how to take risks and not be surprised by reactions to new products: do research & anticipate what people will do.
- When Facebok was a college site, people loved all the features they released. Now the company needs to communicate why a change is good. They give people a choice to opt in now so a new design is not a shock.
- The greatest risk is taking no risk. Small tweaks are safe. Big ideas are what leads to innovation.
More on design at Facebook: Design Lessons From 350 Million outlines how the Facebook design team creates experiences for Facebook's 350 million users.