Why New Ideas Fail

In 2007, Steve Ballmer, then-CEO of Microsoft, emphatically predicted that Apple's new phone would fail. "There's no chance that the iPhone is going to get any significant market share," he said. "No chance."

The volume of Ballmer's voice makes him a popular target in technology, but he wasn't an outlier, just the loudest guy in crowd of skeptical experts. RIM CEO Jim Balsillie said the iPhone would never represent "a sort of sea-change for BlackBerry." Cellphone experts writing in Bloomberg , PC Magazine , and Marketwatch all said it would flop .

No one had seen something like the iPhone before. One large screen? With no keypad? That tries to be everything at once, but actually offers a poor call service, slow Internet speeds, and worse camera quality than your existing devices? The experts were certain: This will not work.

Everybody knows the end of that story. The failure forecasts failed.

To be fair, predicting the future is hard. But what if the industry experts here were wrong about the iPhone, not just because of the uncertainty of predictions, but also because they were experts ?  What if they were blinded by their own knowledge, so confident in what was already working that they couldn't contemplate the feasibility of something new? In 1997, Clayton Christensen coined the term "the Innovator's Dilemma" to describe the choice companies face between incrementally improving their core business (perfecting old ideas) and embracing emerging markets that could upend their core business (investing in new ideas).

But what if the innovator's dilemma is part of something bigger—a creator's dilemma, an innate bias against novelty?

Indeed, it turns out that our aversion to new ideas touches more than technology companies. It affects entertainment executives deciding between new projects, managers choosing between potential projects or employees, and teachers assessing conformist versus non-conformist children. It is a bias against against the new. The brain is hardwired to distrust creativity.

***

The physicist Max Planck put it best: "Science advances one funeral at a time.”

One place to watch the funeral march of science is America's peer-review process for academic research, which allocates $40 billion each year to new ideas in medicine, engineering, and technology. Every year, the National Institutes of Health and the National Science Foundation review nearly 100,000 applications for funding. The vast majority—up to 90 percent in some years—are rejected. For many breakthrough ideas, this selection process is the difference between life and death, financial backing and financial bankruptcy.

What sort of proposals do NIH evaluators approve? It’s a critical question for scientists. And the answer is nobody knows. Submissions receive such widely varying treatment that the relationship between evaluators' decisions is “perilously close to rates found for Rorschach inkblot tests,” according to a 2012 review .

A new ingenious paper raises a dangerous question: Are expert evaluators subtly biased against new ideas?

Researchers Kevin J. Boudreau, Eva Guinan, Karim R. Lakhani, and Christoph Riedl recruited 142 world-class researchers from a leading medical school and randomly assigned them to evaluate several proposals. Sometimes, faculty were experts in the subject of the submissions they read. Often, they were experts in other fields. But in all cases, the experiment was triple-blind: evaluators did not know submitters, submitters did not know evaluators, and evaluators did not talk to each other.

The researchers found that new ideas — those that remixed information in surprising ways — got worse scores from everyone, but they were particularly punished by experts. "Everyone dislikes novelty,” Lakhami explained to me, but “experts tend to be over-critical of proposals in their own domain." Knowledge doesn’t just turn us into critical thinkers. It maybe turns us into over -critical thinkers. (In the real world, everybody has encountered a variety of this: A real or self-proclaimed expert who's impatient with new ideas, because they challenge his ego, piercing the armor of his expertise.)

Experts might be particularly biased against new ideas*, but most people aren't too fond of creativity either, either. In fact, they can be downright hostile.

A 1999 study found that teachers who claim to enjoy creative children don't actually enjoy any of the characteristics associated with creativity, such as non-conformity. A famous 2010 study from the University of Pennsylvania showed that ordinary people often dismiss new ideas, because their uncertainty makes us think, and thinking too hard makes us feel uncomfortable. "People often reject creative ideas even when espousing creativity as a desired goal," the researchers wrote. People are subtly prejudiced against novelty, even when they claim to be open to new ways of thinking.

***

How should creative people fight this widespread prejudice against creativity? Perhaps by disguising their new ideas as old ideas. If people are attracted to the familiar, it’s crucial for creative people to frame their ideas in ways that seem recognizable, predictable, and safe.

We're not prejudiced against all creativity, Karim Lakhani told me. In fact, his team studying academic submissions found that slightly novel medical proposals got the highest ratings. The graph below shows evaluation scores on the Y-axis plotted against the measured novelty of each submission. The overall trajectory is downward. Newer ideas generally got worse ratings. But you'll notice that something important is happening at the left end of the curve ...

Gimme Something New (But Not Too New)

... the line goes up. Indeed, that small bump at the beginning suggests there is an "optimal newness" for ideas that lives somewhere between the fresh and the familiar, Lakhami said.

In Hollywood, the "high-concept pitch" offers a useful example. Film producers, like NIH scientists, have to evaluate hundreds of ideas a year, but can only accept a tiny percentage. To grab their attention, writers often frame original ideas as a fresh combination of existing ideas. "It’s Groundhog Day meets War of the Worlds!” O r “It’s Transformers on the ocean!" In Silicon Valley, where venture capitalists also shift through a surfeit of proposals, the culture of the high-concept pitch is vibrant (Airbnb was once eBay for homes; Uber, Lyft, and Zipcar were all once considered Airbnb for cars; now, people want Uber for everything).

Creative people often bristle at the suggestion that they have to stoop to marketing their ideas. It's more pleasant to think that one's brilliance is self-evident and doesn't require the gloss of sales or the theater of marketing. But whether you're an academic, screenwriter, or entrepreneur, the difference between a brilliant new idea with bad marketing a mediocre idea with excellent marketing can be the difference between success and bankruptcy.

American culture worships creativity, but mostly in the abstract. Most people really don't like new ideas that sound entirely new, particularly the experts that often have to approve them. The trick is learning to frame new ideas as old ideas—to make your creativity seem, well, not quite so creative.

( Image via Peshkova / Shutterstock.com )