Back in 2007, five entrepreneurial firms tried to crack the emerging social investing market. In a multiyear study, Stanford researchers carefully tracked how the startups responded to customer feedback on early, experimental prototypes of their products.
All five firms had the same goal: to marry social networks with personal investing. Amateur investors, they hypothesized, would like an easy way to share tips and advice so they could bypass pricey wealth advisors. Providing a dedicated online social platform that would encourage small investors to engage with each other in a vibrant peer-to-peer mentoring community would be just the ticket.
All the startups started on fairly equal footing. None of them had a clear idea of how they would actually make money. But they all hoped a business model would emerge through leveraging lean startup methods—experimenting with different features and continuously iterating and innovating as they got feedback from customers.
It just happens that the one firm chose the most effective kind of experimentation for that particular market at that particular time.
It did absolutely nothing!
Knowing which method to choose for your particular market can make the difference between success and failure.
That is, these entrepreneurs built a loose and generic social investment platform, and put it out into the world. No continual tweaking of features and functions. No pivots. It just let customers play with its platform and tools, and watched to see what would happen. The feedback from that “experiment” was invaluable. Today, that firm is a leader in the rapidly expanding global social trading and investment market, whereas three of the startups that chose more traditional lean startup experimental methods either failed or went into hibernation.
“This winning firm was thinking in more sophisticated ways about what ‘experimentation’ meant in its specific market,” says Kathleen Eisenhardt. Eisenhardt is the Stanford W. Ascherman M.D. Professor in Stanford University’s Department of Management Science & Engineering and Faculty Co-Director of the Stanford Technology Ventures Program. Eisenhardt’s research sits at the intersection of strategy and organization theory, focusing particularly on high-velocity markets and technology firms.
“As it turned out, the real value of its social platform was connecting amateurs with professional investors, not with each other. Since the other firms rushed ahead with less appropriate experimentation methods, they did not learn the right market insights. This ultimately doomed their efforts.”
Unpacking the meaning of experimentation
In her research, Eisenhardt has identified three different types of experimentation:
- Sequential: A linear, step-by-step, and iterative way of posing hypotheses and letting your customers respond to your product prototypes (like MVPs)
- Parallel: A multi-pronged, simultaneous exploration of different and competing paths forward
- Passive: Sitting back and letting customers figure out what they want from your product or service
Knowing which method to choose for your particular market can make the difference between success and failure. For example, in the social investing market, the early participants had no idea who would want to use such a platform, and for what sorts of activities. There were so many unknowns that responding blindly to early feedback would mean going down multiple rabbit holes—as is what happened to the other startups. In uncertain markets with poor information and many moving parts, leaping into decisive action isn’t always the best course.
Eisenhardt’s research sits at the intersection of strategy and organization theory, focusing particularly on high-velocity markets and technology firms.
Sequential experimentation
The traditional lean experimentation technique is sequential. Startups come up with hypotheses about different features, functions, or uses of their product or service, and then test those hypotheses on customers, one minimally viable product (MVP) after another. Every time they fail, they rinse and repeat until they either hit entrepreneurial gold, or pivot based on what they’ve learned.
“This can be extraordinarily valuable,” says Eisenhardt, who points to Dà-Jiāng Innovations Science and Technology Co. (DJI), an early player in the consumer drone market. DJI went through countless sequential experiments when designing its drone.
A drone is a complex piece of machinery, with multiple engineering challenges to work out: the gimbal, the camera, the aeronautics. DJI decided to create a rough prototype drone that could fly and then focusing on one component at a time—starting with the most important one: the gimbal. After that challenge was resolved, it moved sequentially onto other ones.
“They ended up with an amazing product. And they made it in a very particular, sequential manner, going step-by-step through each of the components,” says Eisenhardt.
When Sequential Works Well:
Sequential works when you need to step through various stages to solve a complex problem. It involves linear uncertainty, where one problem needs to be solved before moving onto the next one.
DJI even went into hyperdrive in its sequential experiments. Instead of assigning just one team to iterate on aeronautical design, it put multiple design teams to work on it, each one iterating sequentially, and failing, and learning and moving onto the next prototype.
This type of experimentation often works well with complex or untried technologies, says Eisenhardt. In the case of the consumer drone market, ensuring that the product could actually fly was obviously the top priority. Everything else followed sequentially from that.
When Sequential Doesn’t Work
But sequential doesn’t always work. One of the social investing startups took the traditional, sequential approach, and failed. It continually experimented and iterated after getting responses from amateur investors. It ended up with a Facebook-like online platform, and added more and more social features with each iteration. As it turned out, a friendship network of amateurs was not the best market. Instead the valuable market was a two-sided marketplace where amateur investors were only one side of the platform. The critical other side of the platform were professional investment advisors who worked out of geographically isolated places like Bend, Oregon, or Telluride, Colorado. Because this startup misunderstood the market opportunity, it built a product for a market that didn’t exist.
Experimentation is an essential tactic for surviving in new and high velocity markets where there are many unknowns.
Parallel experimentation
Parallel experimentation is the opposite of the sequential technique. With the parallel technique, you work on multiple possible solutions simultaneously, and try to cover as much ground as you can. For example, DJI turned to parallel experimentation to find product-market fit. Rather than focusing on one particular market opportunity, DJI sent its people out to multiple trade shows—an Iowa agriculture fair, a real-estate convention and a German sports meeting, for instance. By seeking out and connecting with leading players in each potential vertical, they experimented with how drones might be used in agriculture, real estate, motion picture and other industries.
When Parallel Works
Parallel works when there are multiple alternative paths that might be viable, but you don’t know which one. “We call this the ‘multi-armed bandit’ approach, after the Vegas slot machines,” says Eisenhardt. “Like playing multiple slots at once, pursuing multiple paths at once makes sense if a quick time to market is urgent,” she says.
This was the case for DJI, which had several other drone startups at its heels.
“By doing this parallel experimentation, these entrepreneurs systematically reduced the uncertainty of ‘where will drones really pay off?’ very quickly,” says Eisenhardt. DJI learned almost immediately that the movie industry was willing to pay now—and big—money for drones to replace the airplanes and helicopters that were currently being used for difficult airial shots. The resulting decision to focus on that particular application in that industry paid off for DJI. Today, China-based DJI is a billion dollar company, the global leader in an industry expected to reach $129 billion by 2028.
When Parallel Doesn’t Work
But parallel experimentation can be expensive, because you’re exploring many alternatives simultaneously. And it obviously doesn’t work if you have a series of successive uncertainties to resolve. You have to truly stand at the fork of a multi-pronged highway, with no clear best next step in sight.
And introducing competition into parallel experimentation by pitting teams within your organization against each other has to be used with care. First, it can lead to animosity that can erode your organizational culture. Second, it can lead to knowledge hoarding, as individual teams keep their perhaps-winning “secrets” to themselves.
To mitigate these risks, you can use these implementation tips. One is to keep such competitions short—a few weeks to a few months, not months and years. The other is to switch up the teams after each competition to cross-pollinate ideas.
Passive experimentation
Then there’s the passive experiment: putting your product out there, and waiting for the market to figure it out. “Most lean startups believe that “You need to be always iterating, iterating, iterating, and doing, doing, doing. Well, there are actually times when you want to pause and just wait to see what happens,” says Eisenhardt. “Passivity might be the optimal strategy.”
The most successful social investing company did just that. It built a trading platform, got the SEC approval it needed to operate in the space, and simply put the platform out into the world and let it speak for itself.
“By doing parallel experimentation, entrepreneurs systematically reduced the uncertainty of ‘where will drones really pay off?’ very quickly.”
Kathleen Eisenhardt
When Passive Works
Passive works when you have a lot of unknown unknowns, “that is, you don’t know what you don’t know,” says Eisenhardt. “When there’s a massive amount of ambiguity in the market—as is frequent in emerging spaces—being passive and letting customers figure out for themselves how to use your product or service can give you surprising insights,” says Eisenhardt.
She points to the nascent social investing market as a case in point.
“The winning firm found its customers by just waiting to see who would come onto the platform, and learned from them how they would use it,” says Eisenhardt. Once the startup learned this, it tailored its platform to appeal specifically to professional investors and their match with amateur and other investors. These unexpected insights powered a growth strategy.
When Passive Doesn’t Work
This strategy worked less well for DJI’s competitor, 3D Robotics (3DR), which Eisenhardt also followed for years. 3DR took the passive approach to experimentation by putting its drone technology out into the world, and waited for the proverbial killer app to emerge. It never did. That app never presented itself.
The technology was too new, and would-be customers needed help interpreting its uses other than the “gee-whiz” factor that attracted techy drone hobbyists. The market results speak for themselves: California-based 3DR was forced out of the hardware-based drone business in 2016.
Experimentation works—but figure out the right method for your situation
Experimentation is an essential tactic for surviving in new and high velocity markets where there are many unknowns. In these uncertain markets with many moving parts, leaping into decisive action isn’t always the best course. Research by Eisenhardt and her colleagues will continue to illuminate how best to understand which experimental method suits your particular market, and identify the best strategies for growth.
