Surprise-o-Meter

Source: Poorly Drawn Lines http://www.poorlydrawnlines.com/

Your surprise-o-meter is an internal light bulb that lights up when you encounter new evidence or experiences contrary to your beliefs or assumptions. Epistemic surprises are conveniently valuable for learning because 1. They tend to indicate large shifts in your understanding of the world and 2. They tend to evoke emotional responses, which alerts us to pay attention to what’s causing the surprise. Cashing in on surprising experiences through self reflection can have major positive impact on your ability to learn from experiences.

Even though surprise is considered a universal emotion, how humans deal with the physiological response of surprise varies from person to person.

I’d like to argue against two attitudes towards surprise I see often. The first are people who never appeared surprised. People who never appear surprised signal strong hindsight bias with lower introspection into their epistemology. Faced with new evidence that shapes their understanding of the world, they are capable of quickly changing their model without noticing what has changed. Sometimes worse, they have vague internal models that don’t make specific predictions, and therefore are resistant to contrary data that can be agents of surprise. Expecting the economy to crash in 2016 leaves room for surprises when it doesn’t in 2016. Expecting the economy to crash at some point in the future doesn’t give opportunities for surprises.

Another type of people treat surprise as a fleeting emotion. They say “wow!” and “interesting!” and “that’s so cool!” when they encounter an interesting fact, and then the train of thought ends there. It’s frustrating for me to watch because these people drop the opportunity to extract more insights right when the reflection can begin.

Tuning your surprise-o-meter involves some deliberate practice and some experience. Some low-hanging fruits you can do right now are making more specific predictions (i.e. Make Beliefs Pay Rent (in anticipated experiences)) and bookkeeping them.

So what should you do when you’re surprised? The first thing you should do is try to capture your previous epistemology. Your mental model of the world is about to change, you should capture a snapshot while that understanding is fresh. It’s going to be important for the next part of the exercise.

Afterwards, compare your previous understanding with your current understanding. Highlight the differences and measure how much your beliefs have changed. Think about downstream conclusions or beliefs your new understanding might produce. Recurse your surprise exercise if any of those conclusions are surprising.

When I do this, sometimes I will miss some coverage not covered by the original incident, and capture some more surprise. For example, I might be surprised to find out that cyclists experience a 40% decrease in drag when drafting behind a group. I may uncover other surprises through that initial surprise. “Wait, that makes following a group super important!” -> “Wait, that makes having a group to follow super important!” -> “Wait, that makes it super important to make friends in your race!”.

Now retrace to your old beliefs and notice how the new experience *caused* the change in beliefs. Pay special attention to how this new experience fits into your new mental model, and what new prediction this new model will produce. Sometimes in this stage I will catch that the new information incorrectly updated my belief model, and that is cause for further reflection and interpretation of my experience.

Interleaved in the mental process of reflecting on your surprise, you will review your old state of beliefs. When comparing past and present beliefs, I recall what series of assumptions and evidence has caused me to believe in past belief X.

Afterwards think very hard and carefully of how you could have arrived at your current state of beliefs without the surprising event. This brainstorm constraint is super important because we cannot call upon surprising events to happen anytime we wish. Surprises, by definition, escape our foresight. So if you want to learn how to reach your current beliefs quicker, you can not rely on surprises.

You *can* find ways to reach surprising events earlier, and when you do that, you avoid the trap of relying on surprising to make epistemic progress.

As a taster, some of the things I discovered through this exercise are:

  • “I pattern matched too quickly on a known solution without evaluating how the differences in this particular scenario can change the final solution.” (e.g. thinking that being a good swimmer would automagically make you a good water polo player. Both sports exist on the water, but differences like the end objective and the player interactions outweigh the skills of being a good swimmer).
  • “I should have come up with more adversarial examples when presented with belief X. If I had done that, I would have thought of surprising edgecase Y during that exercise instead of now.”
  • “I heard this from someone else and just assumed it was true. Maybe I shouldn’t have trusted this person’s claims as much.”

Often times when you talk to me, you’ll notice that I would often say something like “oh, that’s interesting! I thought that X was true, but it seems like I didn’t take into account what you said about Y. I should have considered an adversarial example to reach what you said about Y.” Those conversations are fun.

After it all ends, I just mire in the feeling of surprise. It’s a pleasant feeling to bathe in and a wondrous celebration of how wrong I was before. Surprise feels better if you know why you’re surprised.

Summary of exercise:

  1. Notice a surprising event
  2. Capture a snapshot of your previous epistemological state
  3. Compare your current state with your previous state. Highlight any differences
  4. Calculate downstream what conclusions are drawn with your new beliefs
  5. Retrace your old beliefs and notice step-by-step how the event has shifted from your old beliefs to your new ones.
  6. Retrace how you arrived at your old beliefs.
  7. Brainstorm ways on how to reach you could have reached your current belief state without the surprising event.

Surprise is a great self-reflection trigger because a surprising event sheds a lot of light on your epistemology. It is a convenient emotion because humans are hardwired to pay attention to surprising events so you don’t have to seek out events that change your belief states — you will notice them. Now all you have to do when you notice is to learn from these events.

Leave a Reply