Aurora Namnum's Key Ideas from Range
by David Epstein

an influential book on expert judgment was published that Kahneman told me impressed him “enormously.” It was a wide-ranging review of research that rocked psychology because it showed experience simply did not create skill in a wide range of real-world scenarios, from college administrators assessing student potential to psychiatrists predicting patient performance to human resources professionals deciding who will succeed in job training. In those domains, which involved human behavior and where patterns did not clearly repeat, repetition did not cause learning. Chess, golf, and firefighting are exceptions, not the rule.

In 2009, Kahneman and Klein took the unusual step of coauthoring a paper in which they laid out their views and sought common ground. And they found it. Whether or not experience inevitably led to expertise, they agreed, depended entirely on the domain in question. Narrow experience made for better chess and poker players and firefighters, but not for better predictors of financial or political trends, or of how employees or patients would perform. The domains Klein studied, in which instinctive pattern recognition worked powerfully, are what psychologist Robin Hogarth termed “kind” learning environments. Patterns repeat over and over, and feedback is extremely accurate and usually very rapid. In golf or chess, a ball or piece is moved according to rules and within defined boundaries, a consequence is quickly apparent, and similar challenges occur repeatedly. Drive a golf ball, and it either goes too far or not far enough; it slices, hooks, or flies straight. The player observes what happened, attempts to correct the error, tries again, and repeats for years. That is the very definition of deliberate practice, the type identified with both the ten-thousand-hours rule and the rush to early specialization in technical training. The learning environment is kind because a learner improves simply by engaging in the activity and trying to do better. Kahneman was focused on the flip side of kind learning environments; Hogarth called them “wicked.” In wicked domains, the rules of the game are often unclear or incomplete, there may or may not be repetitive patterns and they may not be obvious, and feedback is often delayed, inaccurate, or both.

Moravec’s paradox: machines and humans frequently have opposite strengths and weaknesses.

the game’s strategic complexity provides a lesson: the bigger the picture, the more unique the potential human contribution. Our greatest strength is the exact opposite of narrow specialization. It is the ability to integrate broadly. According to Gary Marcus, a psychology and neural science professor who sold his machine learning company to Uber, “In narrow enough worlds, humans may not have much to contribute much longer. In more open-ended games, I think they certainly will. Not just games, in open ended real-world problems we’re still crushing the machines.”

“AI systems are like savants.” They need stable structures and narrow worlds.

When we know the rules and answers, and they don’t change over time—chess, golf, playing classical music—an argument can be made for savant-like hyperspecialized practice from day one. But those are poor models of most things humans want to learn.

Chris Argyris, who helped create the Yale School of Management, noted the danger of treating the wicked world as if it is kind. He studied high-powered consultants from top business schools for fifteen years, and saw that they did really well on business school problems that were well defined and quickly assessed. But they employed what Argyris called single-loop learning, the kind that favors the first familiar solution that comes to mind. Whenever those solutions went wrong, the consultant usually got defensive. Argyris found their “brittle personalities” particularly surprising given that “the essence of their job is to teach others how to do things differently.”

Flynn conducted a study in which he compared the grade point averages of seniors at one of America’s top state universities, from neuroscience to English majors, to their performance on a test of critical thinking. The test gauged students’ ability to apply fundamental abstract concepts from economics, social and physical sciences, and logic to common, real-world scenarios. Flynn was bemused to find that the correlation between the test of broad conceptual thinking and GPA was about zero. In Flynn’s words, “the traits that earn good grades at [the university] do not include critical ability of any broad significance.”

The overall experiment results went like this: the more hints that were available during training, the better the monkeys performed during early practice, and the worse they performed on test day. For the lists that Macduff spent three days practicing with automatic hints, he got zero correct. It was as if the pair had suddenly unlearned every list that they practiced with hints. The study conclusion was simple: “training with hints did not produce any lasting learning.”

If that eighth-grade classroom followed a typical academic plan over the course of the year, it is precisely the opposite of what science recommends for durable learning—one topic was probably confined to one week and another to the next. Like a lot of professional development efforts, each particular concept or skill gets a short period of intense focus, and then on to the next thing, never to return. That structure makes intuitive sense, but it forgoes another important desirable difficulty: “spacing,” or distributed practice. It is what it sounds like—leaving time between practice sessions for the same material. You might call it deliberate not-practicing between bouts of deliberate practice.

In 2007, the U.S. Department of Education published a report by six scientists and an accomplished teacher who were asked to identify learning strategies that truly have scientific backing. Spacing, testing, and using making-connections questions were on the extremely short list. All three impair performance in the short term.

“Above all, the most basic message is that teachers and students must avoid interpreting current performance as learning. Good performance on a test during the learning process can indicate mastery, but learners and teachers need to be aware that such performance will often index, instead, fast but fleeting progress.”

the students settled into a worksheet for what psychologists call “blocked” practice. That is, practicing the same thing repeatedly, each problem employing the same procedure. It leads to excellent immediate performance, but for knowledge to be flexible, it should be learned under varied conditions, an approach called varied or mixed practice, or, to researchers, “interleaving.” Interleaving has been shown to improve inductive reasoning. When presented with different examples mixed together, students learn to create abstract generalizations that allow them to apply what they learned to material they have never encountered before.

In a study using college math problems, students who learned in blocks—all examples of a particular type of problem at once—performed a lot worse come test time than students who studied the exact same problems but all mixed up. The blocked-practice students learned procedures for each type of problem through repetition. The mixed-practice students learned how to differentiate types of problems.

In one of Kornell and Bjork’s interleaving studies, 80 percent of students were sure they had learned better with blocked than mixed practice, whereas 80 percent performed in a manner that proved the opposite. The feeling of learning, it turns out, is based on before-your-eyes progress, while deep learning is not. “When your intuition says block,” Kornell told me, “you should probably interleave.”

The “desirable difficulty” coiner himself, Robert Bjork, once commented on Shaquille O’Neal’s perpetual free-throw woes to say that instead of continuing to practice from the free-throw line, O’Neal should practice from a foot in front of and behind it to learn the motor modulation he needed.

If you’re asked to predict whether a particular horse will win a race or a particular politician will win an election, the more internal details you learn about any particular scenario—physical qualities of the specific horse, the background and strategy of the particular politician—the more likely you are to say that the scenario you are investigating will occur.

There was a group of students, however, who were particularly good at finding common deep structures: students who had taken classes in a range of domains, like those in the Integrated Science Program. Northwestern’s website for the program features an alum’s description: “Think of the Integrated Science Program as a biology minor, chemistry minor, physics minor, and math minor combined into a single major. The primary intent of this program is to expose students to all fields of the natural and mathematical sciences so that they can see commonalities among different fields of the natural sciences. . . . The ISP major allows you to see connections across different disciplines.”

successful problem solvers are more able to determine the deep structure of a problem before they proceed to match a strategy to it. Less successful problem solvers are more like most students in the Ambiguous Sorting Task: they mentally classify problems only by superficial, overtly stated features, like the domain context. For the best performers, they wrote, problem solving “begins with the typing of the problem.” As education pioneer John Dewey put it in Logic, The Theory of Inquiry, “a problem well put is half-solved.”

In the face of the unexpected, the range of available analogies helped determine who learned something new. In the lone lab that did not make any new findings during Dunbar’s project, everyone had similar and highly specialized backgrounds, and analogies were almost never used. “When all the members of the laboratory have the same knowledge at their disposal, then when a problem arises, a group of similar minded individuals will not provide more information to make analogies than a single individual,” Dunbar concluded. “It’s sort of like the stock market,” he told me. “You need a mixture of strategies.”

“Match quality” is a term economists use to describe the degree of fit between the work someone does and who they are—their abilities and proclivities.

“The benefits to increased match quality . . . outweigh the greater loss in skills.” Learning stuff was less important than learning about oneself. Exploration is not just a whimsical luxury of education; it is a central benefit.

“admonitions such as ‘winners never quit and quitters never win,’ while well-meaning, may actually be extremely poor advice.” Levitt identified one of his own most important skills as “the willingness to jettison” a project or an entire area of study for a better fit. Winston Churchill’s “never give in, never, never, never, never” is an oft-quoted trope. The end of the sentence is always left out: “except to convictions of honor and good sense.”

Seth Godin, author of some of the most popular career writing in the world, wrote a book disparaging the idea that “quitters never win.” Godin argued that “winners”—he generally meant individuals who reach the apex of their domain—quit fast and often when they detect that a plan is not the best fit, and do not feel bad about it. “We fail,” he wrote, when we stick with “tasks we don’t have the guts to quit.” Godin clearly did not advocate quitting simply because a pursuit is difficult. Persevering through difficulty is a competitive advantage for any traveler of a long road, but he suggested that knowing when to quit is such a big strategic advantage that every single person, before undertaking an endeavor, should enumerate conditions under which they should quit. The important trick, he said, is staying attuned to whether switching is simply a failure of perseverance, or astute recognition that better matches are available.

On the final weekend of the 2018 Winter Olympics, Sasha Cohen, a 2006 silver medalist figure skater, wrote an advice column to retiring athletes. “Olympic athletes need to understand that the rules for life are different from the rules for sports,” she wrote. “Yes, striving to accomplish a single overarching goal every day means you have grit, determination and resilience. But the ability to pull yourself together mentally and physically in competition is different from the new challenges that await you. So after you retire, travel, write a poem, try to start your own business, stay out a little too late, devote time to something that doesn’t have a clear end goal.” In the wider world of work, finding a goal with high match quality in the first place is the greater challenge, and persistence for the sake of persistence can get in the way.

85 percent were either “not engaged” with their work or “actively disengaged.” In that condition, according to Seth Godin, quitting takes a lot more guts than continuing to be carried along like debris on an ocean wave. The trouble, Godin noted, is that humans are bedeviled by the “sunk cost fallacy.” Having invested time or money in something, we are loath to leave it, because that would mean we had wasted our time or money, even though it is already gone. Writer, psychology PhD, and professional poker player Maria Konnikova explained in her book The Confidence Game how the sunk cost mindset is so deeply entrenched that conmen know to begin by asking their marks for several small favors or investments before progressing to large asks. Once a mark has invested energy or money, rather than walking away from sunk costs he will continue investing, more than he ever wanted to, even as, to any rational observer, disaster becomes imminent. “The more we have invested and even lost,” Konnikova wrote, “the longer we will persist in insisting it will all work out.”

Career goals that once felt safe and certain can appear ludicrous, to use Darwin’s adjective, when examined in the light of more self-knowledge. Our work preferences and our life preferences do not stay the same, because we do not stay the same. Psychologist Dan Gilbert called it the “end of history illusion.” From teenagers to senior citizens, we recognize that our desires and motivations sure changed a lot in the past (see: your old hairstyle), but believe they will not change much in the future. In Gilbert’s terms, we are works in progress claiming to be finished.

Instead of asking whether someone is gritty, we should ask when they are. “If you get someone into a context that suits them,” Ogas said, “they’ll more likely work hard and it will look like grit from the outside.”

Paul Graham, computer scientist and cofounder of Y Combinator—the start-up funder of Airbnb, Dropbox, Stripe, and Twitch—encapsulated Ibarra’s tenets in a high school graduation speech he wrote, but never delivered: It might seem that nothing would be easier than deciding what you like, but it turns out to be hard, partly because it’s hard to get an accurate picture of most jobs. . . . Most of the work I’ve done in the last ten years didn’t exist when I was in high school. . . . In such a world it’s not a good idea to have fixed plans. And yet every May, speakers all over the country fire up the Standard Graduation Speech, the theme of which is: don’t give up on your dreams. I know what they mean, but this is a bad way to put it, because it implies you’re supposed to be bound by some plan you made early on. The computer world has a name for this: premature optimization. . . . . . . Instead of working back from a goal, work forward from promising situations. This is what most successful people actually do anyway. In the graduation-speech approach, you decide where you want to be in twenty years, and then ask: what should I do now to get there? I propose instead that you don’t commit to anything in the future, but just look at the options available now, and choose those that will give you the most promising range of options afterward.

“the further the problem was from the solver’s expertise, the more likely they were to solve it.”

“The shortcut [for a lack of ideas] is competition in the realm of computing power,” Yokoi explained. “When it comes to that . . . the screen manufacturers and expert graphics designers come out on top. Then Nintendo’s reason for existence disappears.” He felt that the lateral and vertical thinkers were best together, even in highly technical fields. Eminent physicist and mathematician Freeman Dyson styled it this way: we need both focused frogs and visionary birds. “Birds fly high in the air and survey broad vistas of mathematics out to the far horizon,” Dyson wrote in 2009. “They delight in concepts that unify our thinking and bring together diverse problems from different parts of the landscape. Frogs live in the mud below and see only the flowers that grow nearby. They delight in the details of particular objects, and they solve problems one at a time.” As a mathematician, Dyson labeled himself a frog, but contended, “It is stupid to claim that birds are better than frogs because they see farther, or that frogs are better than birds because they see deeper.” The world, he wrote, is both broad and deep. “We need birds and frogs working together to explore it.” Dyson’s concern was that science is increasingly overflowing with frogs, trained only in a narrow specialty and unable to change as science itself does. “This is a hazardous situation,” he warned, “for the young people and also for the future of science.”

“If you’re working on well-defined and well-understood problems, specialists work very, very well,” he told me. “As ambiguity and uncertainty increases, which is the norm with systems problems, breadth becomes increasingly important.”

He learned that specializing in a topic frequently did not bear fruit in the forecasts. “So if I know somebody [on the team] is a subject area expert, I am very, very happy to have access to them, in terms of asking questions and seeing what they dig up. But I’m not going to just say, ‘Okay, the biochemist said a certain drug is likely to come to market, so he must be right.’ Often if you’re too much of an insider, it’s hard to get good perspective.” Eastman described the core trait of the best forecasters to me as: “genuinely curious about, well, really everything.”

Narrow experts are an invaluable resource, she told me, “but you have to understand that they may have blinders on. So what I try to do is take facts from them, not opinions.”

Just as Tetlock says of the best forecasters, it is not what they think, but how they think. The best forecasters are high in active open-mindedness. They are also extremely curious, and don’t merely consider contrary ideas, they proactively cross disciplines looking for them. “Depth can be inadequate without breadth,” wrote Jonathan Baron, the psychologist who developed measurements of active open-mindedness.

forecasters can improve by generating a list of separate events with deep structural similarities, rather than focusing only on internal details of the specific event in question. Few events are 100 percent novel—uniqueness is a matter of degree, as Tetlock puts it—and creating the list forces a forecaster implicitly to think like a statistician.

For example, in 2015, forecasters were asked if Greece would exit the eurozone that year. No country had ever left, so the question seemed totally unique. But there were plenty of examples of international negotiation failures, exits from international agreements, and forced currency conversions that allowed the best forecasters to ground themselves in what usually happens without focusing narrowly on all the unique details of the present situation. Starting with the details—the inside view—is dangerous. Hedgehog experts have more than enough knowledge about the minutiae of an issue in their specialty to do just what Dan Kahan suggested: cherry-pick details that fit their all-encompassing theories. Their deep knowledge works against them. Skillful forecasters depart from the problem at hand to consider completely unrelated events with structural commonalities rather than relying on intuition based on personal experience or a single area of expertise.

In Tetlock’s twenty-year study, both foxes and hedgehogs were quick to update their beliefs after successful predictions, by reinforcing them even more strongly. When an outcome took them by surprise, however, foxes were much more likely to adjust their ideas. Hedgehogs barely budged. Some hedgehogs made authoritative predictions that turned out wildly wrong, and then updated their theories in the wrong direction. They became even more convinced of the original beliefs that led them astray. “Good judges are good belief updaters,” according to Tetlock. If they make a bet and lose, they embrace the logic of a loss just as they would the reinforcement of a win. That is called, in a word: learning. Sometimes, it involves putting experience aside entirely.

In four separate fires in the 1990s, twenty-three elite wildland firefighters refused orders to drop their tools and perished beside them. Even when Rhoades eventually dropped his chainsaw, he felt like he was doing something unnatural. Weick found similar phenomena in Navy seamen who ignored orders to remove steel-toed shoes when abandoning a ship, and drowned or punched holes in life rafts; fighter pilots in disabled planes refusing orders to eject;

“Dropping one’s tools is a proxy for unlearning, for adaptation, for flexibility,” Weick wrote. “It is the very unwillingness of people to drop their tools that turns some of these dramas into tragedies.”

Compare yourself to yourself yesterday, not to younger people who aren’t you. Everyone progresses at a different rate, so don’t let anyone else make you feel behind. You probably don’t even know where exactly you’re going, so feeling behind doesn’t help.

He emphasized that there is a difference between the chain of command and the chain of communication, and that the difference represents a healthy cross-pressure. “I warned them, I’m going to communicate with all levels of the organization down to the shop floor, and you can’t feel suspicious or paranoid about that,” he said. “I told them I will not intercept your decisions that belong in your chain of command, but I will give and receive information anywhere in the organization, at any time. I just can’t get enough understanding of the organization from listening to the voices at the top.”

Geveden joined NASA in 1990, and was a keen observer of the culture. “When I was coming through NASA,” he said, “I had the intuition that there’s a real conformance culture.” Early in his tenure, he attended a team-building class offered by the agency. On the very first day the instructor asked the class, rhetorically, for the single most important principle in decision making. His answer: to get consensus. “And I said, ‘I don’t think the people who launched the space shuttle Challenger agree with that point,’” Geveden told me. “Consensus is nice to have, but we shouldn’t be optimizing happiness, we should be optimizing our decisions. I just had a feeling all along that there was something wrong with the culture. We didn’t have a healthy tension in the system.” NASA still had its hallowed process, and Geveden saw everywhere a collective culture that nudged conflict into darkened corners. “You almost couldn’t go into a meeting without someone saying, ‘Let’s take that offline,’” he recalled, just as Morton Thiokol had done for the infamous offline caucus.

Roger Boisjoly’s unquantifiable argument that the cold weather was “away from goodness” was considered an emotional argument in NASA culture. It was based on interpretation of a photograph. It did not conform to the usual quantitative standards, so it was deemed inadmissible evidence and disregarded. The can-do attitude among the rocket-booster group, Vaughan observed, “was grounded in conformity.” After the tragedy, it emerged that other engineers on the teleconference agreed with Boisjoly, but knew they could not muster quantitative arguments, so they remained silent. Their silence was taken as consent. As one engineer who was on the Challenger conference call later said, "If I feel like I don't have data to back me up, the boss's opinion is better than mine."

When it became utterly ludicrous to carry the saw further, Rhoades still “could not believe” he was parting with it. He felt naked, just as Larry Mulloy said he would have without a quantitative argument for a last-second launch reversal. At NASA, accepting a qualitative argument was like being told to forget you are an engineer. When sociologist Diane Vaughan interviewed NASA and Thiokol engineers who had worked on the rocket boosters, she found that NASA’s own famous can-do culture manifested as a belief that everything would be fine because “we followed every procedure”; because “the [flight readiness review] process is aggressive and adversarial”; because “we went by the book.” NASA’s tools were its familiar procedures. The rules had always worked before. But with Challenger they were outside their usual bounds, where "can do" should have been swapped with what Weich calls a "make do" culture. They needed to improvise rather than throw out information that did not fit the established rubric.

For wildland firefighters, their tools are what they know best. “Firefighting tools define the firefighter’s group membership, they are the firefighter’s reason for being deployed in the first place,” Weick wrote. “Given the central role of tools in defining the essence of a firefighter, it is not surprising that dropping one’s tools creates an existential crisis.” As Maclean succinctly put it, "When a firefighter is told to drip his firefighting tools, he is told to forget being a firefighter."

Rather than adapting to unfamiliar situations, whether airline accidents or fire tragedies, Weick saw that experienced groups became rigid under pressuew and "regress to what they know best."