Wisdom Hill » Power of Experiments & The Right It

Ever wondered why some ideas skyrocket to success while others, despite immense effort, fall flat? In a world saturated with data, how do we move beyond mere intuition and make decisions that truly work? Today, we explore the powerful role of experiments – not just in science, but in business and policy. Specifically, we'll focus on how they can help us identify "The Right It" for new ventures before it's too late. This journey is guided by insights from two key books: "The Power of Experiments" and "The Right It."

Part 1: The Universal Power of Experiments

Experiments, particularly Randomized Controlled Trials (RCTs), have moved beyond the lab to become fundamental tools across diverse fields, including business and government. The core idea is simple yet potent: by randomly assigning participants to different conditions, we can isolate the precise effect of a specific intervention and establish clear cause-and-effect relationships, rather than relying on guesswork or intuition.

While traditionally linked to medicine and the natural sciences, experimental methodology gained further traction in the social sciences, especially psychology, through behavioral experiments. Experiments are crucial for understanding if a particular action can improve decision-making in a specific context. However, translating results from a controlled lab setting to the complex real world is challenging. As the saying goes, "context matters, and no two real-world settings are identical." Lab results might indicate the possibility of an effect, but they can't precisely predict its magnitude or outcome in a specific real-world situation.

Tech companies were early and enthusiastic adopters of this experimental approach. Their massive user bases, the ease of random assignment, and the facility of online data collection significantly lowered traditional barriers to experimentation, such as participant scarcity, randomization difficulties, and data collection limitations. Google, for instance, ran over 10,000 experiments in 2018 alone. This shift from decisions based on fervent debate or managerial intuition to evidence-based decision-making is a key lesson from the tech sector.

Let's look at some striking examples of how experiments have challenged assumptions and guided better decisions:

eBay's $50 Million Advertising Misstep: eBay was spending roughly $50 million annually on Google ads for search terms that included "eBay." Superficially, this seemed like a successful investment, as users clicking these ads often made purchases. However, economists at eBay suspected that users searching for "eBay" would likely have found their way to the site anyway. They ran experiments, turning ads on and off in different markets and tracking traffic changes. The results were startling: when paid ads were off, organic traffic spiked. Loyal customers were simply clicking the top result (the ad) when available, but would scroll down to the organic link otherwise. This experiment revealed that a significant portion of their ad spend was ineffective, saving the company millions. It clearly demonstrated that correlation does not equal causation.

Alibaba's Deep Discount Experiment: Alibaba experimented with offering large discounts on items customers left in their shopping carts for over 24 hours. A well-designed experiment showed that users who received discounts were more likely to buy those discounted items in the short term. However, tracking long-term metrics revealed that overall user spending on the platform did not increase. Users had learned to game the system by adding items to their carts and waiting for discounts. This case underscores the critical need to focus on long-term consequences, not just short-term gains, as the value of an experiment is limited by the outcomes you are able to measure.

Airbnb's Racial Discrimination Experiment: Experiments can also be powerful tools for addressing social issues. Unlike anonymous booking sites, Airbnb's profiles show guest names and pictures, giving hosts the power to accept or reject bookings. Researchers created fake guest profiles with names common to either White or African American individuals and sent booking inquiries. The experiment revealed that profiles with African American-sounding names were 16% less likely to receive acceptance from hosts, even from hosts with no prior experience hosting Black guests. This data provided clear evidence of discrimination, prompting Airbnb to implement changes like incentivizing instant bookings to address the issue.

These examples clearly show how easily our intuition can be flawed, and how experiments provide the objective evidence needed to make better decisions. Experiments also play a vital role in tackling social issues and informing policymaking. Transparency in the experimentation process is crucial for building trust with users. Ultimately, experiments are becoming an indispensable part of a leader's toolkit for continuous learning and adaptation in a data-driven world.

Part 2: Finding "The Right It" – Experimentation in New Business Success

While experiments are invaluable for optimizing existing products and policies, their importance is absolute when planning something entirely new. Alberto Savoia, author of "The Right It," highlights the harsh reality that up to 90% of new products, services, and businesses fail soon after launch. This is usually not because they are poorly built, but because they are the "Wrong It" – an idea the market simply doesn't want. Competent execution alone cannot save a poorly conceived product.

The problem often begins in what Savoia calls "Thoughtland" – an imaginary space where ideas exist as abstract concepts, attracting subjective opinions and predictions without real-world validation. Relying on opinions (yours, others', or even experts') or "Other People's Data" (OPD) from different contexts is highly risky and often leads to investing heavily in the "Wrong It."

To find "The Right It," you must escape Thoughtland and collect Your Own Data (Yoda) directly from your target market. The most valuable Yoda comes with "skin in the game" – a real commitment from potential customers, such as their time, reputation, or, most importantly, money. Mere opinions or 'likes' have zero skin in the game. Quality data (with skin in the game) trumps quantity data (without it).

"The Right It" offers a structured approach to validating your idea through experiments, moving from vague concepts to concrete data:

Thinking Tools: First, clarify your vague idea into a precise, testable Market Engagement Hypothesis – a short statement about how the market will engage with your product. Then, quantify it using the XYZ Hypothesis: "At least X% of Y will do Z" (for instance, "At least 10% of people who live in cities with an Air Quality Index greater than 100 will buy a $120 portable pollution sensor"). You can then use Hypozooming to focus on a smaller, more accessible market subset for initial testing.

Pretotyping Tools: This is where you test your XYZ hypothesis rapidly and inexpensively before building the actual product. Pretotyping is different from traditional prototyping, which tests if you can build it. Pretotyping is about "pretending to have" the product to see if the market wants it – if it's worth building. Key techniques include:

Fake Door: Setting up an ad or website for a product that doesn't exist to see how many people express interest or try to buy.
Pinocchio: Using a non-functional mockup to test usability and desirability.
Mechanical Turk: Simulating a technology's function with a hidden human operator to test user interaction (like IBM's speech-to-text experiment).
One-Night Stand: Offering the product or service for a very limited time or instance (like Airbnb's first rentals).Many other techniques exist, all focused on collecting Yoda quickly and cheaply.

Analysis Tools: Once you have Yoda, analyze it objectively. The Skin-in-the-Game Caliper helps quantify the level of commitment in your data. The TRI Meter (The Right It Meter) is a visual tool to estimate the likelihood of market success based on the strength of your experimental results, helping you decide whether to proceed, tweak the idea, or abandon it. It's important to run multiple experiments to build confidence, especially before making significant investments.

To execute these tests efficiently, Savoia proposes "Plastic Tactics," emphasizing flexibility and speed:

Think Globally, Test Locally: Start testing your idea on a small, accessible market subset before scaling up.
Testing Now Beats Testing Later: Get your idea out of Thoughtland and into the real world as quickly as possible.
Think Cheap, Cheaper, Cheapest: Find the most inexpensive ways to gather reliable data.
Tweak It and Flip It Before You Quit It: If initial results are discouraging, try modifying the idea or flipping assumptions and testing again before giving up.

This iterative process of hypothesizing, experimenting, collecting Yoda, analyzing results, and making decisions is the core of validating a new idea. Diligently applying these tools and tactics can drastically increase your odds of finding "The Right It" and avoiding the costly failure of building something nobody wants.

Conclusion

From uncovering surprising advertising inefficiencies and social biases to fundamentally rethinking how to launch new products, experiments are indispensable. They are not just for scientists or large corporations; they are powerful tools for anyone making decisions in an uncertain world, particularly entrepreneurs and innovators. By embracing experimentation, collecting your own data with "skin in the game," and staying flexible, you can navigate the unpredictable market with greater confidence and build something that truly resonates.