In the tech community, product leaders love sharing their ‘Aha! Moment.’ The zap of sudden insight and discovery. The moment when the value of your product resonates with your users.
While the stories of “Aha! Moments’” seem like a dime a dozen, the journey is far more complex than it appears.
Take, for example, Facebook’s “Aha! Moment,” when they realized that the key to user loyalty boiled down to new users earning 7 friends in their first ten days. The data sounds straightforward, but this lightning bolt of discovery didn’t just happen overnight. It was a multi-year process of intuitive judgment, exemplary people management—and yes, data analysis—that created this “Aha! moment.”
The tale of “7 friends in 10 days” became an overly simplistic rallying cry for the idea that startups should be data-driven, rather than relying on intuition, but this interpretation glosses over some of the crucial facts of the story.
It's better to use data than to not, of course. But at Scuba, we prefer to talk about being data-informed, rather than data-driven. The difference isn't just semantic. Read on to find out why.
The difference between driven and informed
Every massively successful tech company from Google to Uber has found its competitive advantage in data. From the tracking of fundamental metrics like revenue and burn rate to KPIs and success metrics that vary between teams, organizations today need data to guide decision-making.
That's table stakes. The thing that separates great companies from the rest, however, lies in how much of the thinking they do themselves and how much of it is left to the data.
This is the fundamental difference between being data-driven and being data-informed:
- Data-driven: You let the data guide your decision-making process.
- Data-informed: You let data act as a check on your intuition.
When data guides your decision-making, you're always collecting more of it to have the most accurate models and find trends. But, here's the catch—data is never going to tell you the full story, and it's never going to tell you exactly what you need to do next to keep growing.
Too many companies think data should be their only guide and wind up over-optimizing small things rather than developing a bigger product vision.
Being data-informed means using both intuition and data to come up with testable hypotheses about your product. The qualitative complement the quantitative, and vice versa. Data is part of a larger context, one which acknowledges the possibility of bias in both its collection and analysis.
People assume that the process at Facebook was totally data-driven—that “7 friends in 10 days” was a capital-T truth, which after being discovered, set the agenda for the entire company. It's thought that it was part of the logic of Facebook itself, a physical law underlying its growth, and once it was found, Facebook's fate was all but sealed.
The reality is much messier. The work at Facebook was data-informed—not data-driven—in that the key insights came from the intuition of people who knew the product and its users.
Why you need to be data-informed
The early growth team at Facebook got started by closely observing those users who stuck with Facebook and kept using it over the long term. While looking for events correlated with that outcome, they stumbled across a connection between early friend acquisition and retention.
It wasn't causal, but there was a plausible case for a link—Facebook was all about connecting to your friends, so it stood to reason that users who connected to more friends would see the value in Facebook (and stick around). It was the best metric they had, so they went for it. Management got everyone rallied around the idea of getting users to “7 friends in 10 days” and started tracking the team's progress. They “cut it by different dimensions and made lots of hypotheses,” investing time into tooling and measurement to both make changes quickly and see the effects of those changes.
There was no lightning bolt, just an observed correlation that provided a signpost for the entire company to follow. As long as they monitored their progress with data, they could know they were growing.
But the “7 friends in 10 days” metric isn't what told Facebook which experiments to run on their way to a billion users and beyond. They knew what to do because of their great intuition for the product and its users.
“Knowing true product value allows you to design the experiments necessary so that you can really isolate cause and effect,” Chamath Palihapitiya said of Facebook's early growth and experimentation.
How data-informed experimentation works
When Airbnb first launched, it displayed listings like this:
As the site grew in popularity, Airbnb redesigned the search results page like this:
It's not hard to argue that the second version looks nicer than the original—it makes greater use of listing photos, expands the map for easier searching, and minimizes the influence of aspects like the copy attached to each listing.
When they analyzed the results of the rollout, they found that the new design was actually performing worse than the old one.
Of its UI iterative process, Airbnb notes, it's valuable to have a single global metric that you assess experiments against—it keeps you from looking for results that validate your biases by “cherry-picking” the right evidence. This case, however, demanded deeper context and more information—the results were that strange.
When the Airbnb team looked into the test further, they found that the new design actually was performing better—as long as you weren't looking at the site on Internet Explorer. On IE, some crucial elements of the website were broken by the new design. Once fixed, the data started to reflect improved performance on Airbnb's core metric.
As Airbnb continues to mature and collect more customer insights, they continue to evolve their platform. The latest design looks like this:
Who needs to be data-informed?
Being data-informed is critical for Airbnb as they continue to redesign its interface to improve the user experience. Data could not generate the idea for the redesign in the first place, nor could it (on its own) make clear why it initially seemed to perform poorly. Human guidance was needed for that. Data was there merely to help.
This doesn't mean you can relegate all data-related tasks to a single product manager or data scientist. On the contrary, it makes it even more important that every member of your team be fluent with data.
You need to be data-informed when you're first interviewing customers and trying to get a better feel for the problem your company solves. Scoring your potential customers along dimensions of enthusiasm, intent, and use case isn't about turning customer development into a soulless process—it's about coming away with easily digestible insights.
You need to be data-informed when doing customer support. Without metrics that incentivize the members of your support team to focus on the customer, they'll focus on getting good ratings, answering quickly, or finding the easiest tickets.
You need to be data-informed when doing design and UX. Data will never be able to tell you what your product should look and feel like, but it can help you assess whether a particular design is helping your users reach their goals or not.
When your organization is data-informed, everyone in it can derive value from data. Want to learn more about becoming a data-informed organization?
See how Scuba Analytics can help every member of your team become data fluent, to make your organization data-informed.
