Episode 7: Finding Things Out
“Conversation with the MythBusters (Part 1) … Imperfect predictors … The drift toward mediocrity … The ebb and flow of random events … Smoothing out the lumps … Small classes are better … 6 leads of opinion change … Conversation with the MythBusters (Part 2) … The story so far”
Summaries
- Episode 7 - Finding Things Out > Conversation with the MythBusters (Part 1)
- Episode 7 - Finding Things Out > Imperfect predictors > Imperfect predictors
- Episode 7 - Finding Things Out > The drift toward mediocrity > The drift toward mediocrity
- Episode 7 - Finding Things Out > The ebb and flow of random events > The ebb and flow of random events
- Episode 7 - Finding Things Out > Small classes are better > Small classes are better
- Episode 7 - Finding Things Out > Conversation with the MythBusters (Part 2) > Conversation with the MythBusters (Part 2)
- Episode 7 - Finding Things Out > 6 leads of opinion change > 6 leads of opinion change
- Episode 7 - Finding Things Out > The story so far > The story so far
Episode 7 – Finding Things Out > Conversation with the MythBusters (Part 1)
- It turns out there’s nothing, no evidence for that whatsoever.
- Another one is emergency room nurses claim that there’s a lot more activity when there’s a full moon during regular days of the month.
- These are superstitious beliefs as a result of seeing that two things tend to be linked: a pair of lucky socks and how they perform, or the full moon and the activity in that evening, but there is no link.
- There can’t be any link between them, but people see certainly that there is a link.
- They’re familiar with the idea that if we want to believe something, we’ll go and seek out evidence for it and we won’t seek out evidence against it.
- It’s a very pronounced tendency to treat information that’s consistent with what we want to believe in a pretty friendly way and be really hostile to information that’s consistent with something we don’t want to believe.
- It’s almost as if we ask ourselves of something that we want to believe: can I believe this, or is there evidence for this? There’s evidence for almost anything.
- Even the most outlandish things, there is some evidence for it.
- The question is: is there enough evidence? Is there sufficient evidence? We don’t tend to ask ourselves: must I believe this? Is there enough evidence here? So all of that’s true.
- All of people can relate to that, but it’s even more pronounced than that-that is, even if you don’t care about a particular belief, you have no vested interest in it, you tend to look for evidence consistent with the idea rather than information that’s inconsistent with it, which, of course, if we want to have a balanced picture, we’ve got to look at both.
- How would you test that? Well, if you’re like most people, you’d give it a lot of water and see how they do.
- You look for evidence for it rather than against it.
- At some level, it make sense because it reflects a broader belief that, “Look, if this thing’s true, there must be some evidence for it, so let me look for some evidence for it.” You’re doing a very reasonable thing.
- You need to look not only for evidence for something but evidence against it.
- So if you believe that cheerful people are more likely to overcome a bout of cancer, you need to look not just who are the cheerful people you know who’ve done very well, but maybe you know some dour people also who’ve recovered.
- We spoke a lot about how we can contest claims, how it is that we can convince ourselves and others that there’s a real effect here.
Episode 7 – Finding Things Out > Imperfect predictors > Imperfect predictors
- One way of doing this, I think, is looking at this idea of bistable images or ambiguous figures.
- Now if we take the Necker cube, this can be interpreted in one of two ways: you can look at the Necker cube and see the front face as either being on the lower left, or the front face as being in the upper right.
- These are the two ways of seeing a Necker cube, and you can’t see them both at the same time, obviously.
- Another figure is called the old woman or young woman, depending on how you’re looking at it.
- So it’s either an old woman kind of looking forward or looking down, or a young woman who’s looking back.
- It’s often accompanied by a statement about personality, like-people who are right brained see her spinning clockwise; people who are left brained see her spinning anticlockwise, whatever that means, I don’t know about left brain versus right brain.
- Most people see her spinning clockwise, but it takes a little bit of effort for people who, if they’re watching this, to spend a bit of time looking at this spinning dancer and see if they can get her to reverse.
- Now if you take a more realistic figure, for example-say, a picture of your mother or of your partner-if you’re looking at this two-dimensional photograph of your mother or partner, depending on who’s looking at it, you encounter very different experiences.
- You have a lifetime of experience looking at a photograph of your mother.
- When you’re looking at that photo, your heart rate might increase.
- You might have a few memories that come to mind-very different from a complete stranger who’s looking at that particular face.
- If we take even-it goes a little bit more abstract in looking at this type of things-take an abstract concept like promotion.
- The way that you look at that promotion is going to be very different from how your partner looks at that promotion, or your boss looks at that promotion, or your colleagues look at that promotion.
- That interpretation-or in predicting whether to promote somebody-that’s a completely different frame depending on who’s looking at it, who’s doing the interpretation.
- These are the same problems that scientists have to navigate, particularly psychologists-cognitive psychologists-when trying to predict people’s behavior in the future.
- The light physicist Richard Feynman talks about this in a series called “The Pleasure of Finding Things Out.” Now you’re talking about predicting human behavior, but let’s say we’re trying to predict where something will land when we drop it.
- Now if you scale that up just a little bit and predict where feather will land if I drop it on the floor, think about the factors that are interacting with that feather that will determine where it lands.
- Now scale up from predicting where that feather will land to predicting what this bag of cells will do tomorrow-to predicting human behavior.
- There are millions, billions, trillions of different factors that will determine what we will do, and we’re trying to predict that behavior in the future.
- You’re reducing the ways that-you’re reducing the number of variables that can have an effect and so hopefully increasing your ability to predict that behavior in the future.
- We’re going to talk about this idea of multiple predictors: all of these interacting complex factors that we’re trying to constrain, trying to overcome in predicting future behavior, future events, and that kind of leads to things like magical thinking, superstition, as we’ll see next.
Episode 7 – Finding Things Out > The drift toward mediocrity > The drift toward mediocrity
- Now on the particular day, there’s going to be some things working for you and some things working against you.
- These are the things that are going to have a really large impact on how well you impress that interview panel and whether you get the job or not.
- Those things are-I’m fairly confident that they are independent.
- These factors are going to have much more of an impact.
- They’re going to have a large impact on how well you perform in that interview.
- It comes down to the fact that on any sort of event, some things are going to work in your favor.
- It feels-again, imagine, sometimes it feels like that, right? Things are conspiring against you in the world.
- It feels like the world is ganging up on you and everything is just going really badly.
- On the other hand, some things that just feel-you know, on top of your game, and you have soundtrack playing in your head; you’re walking around; it’s fantastic.
- Sometimes these things-sometimes there are runs of good things.
- Sometimes there’s runs of bad things, but they end.
- The observable thing is exactly as we’ve described.
- These lumpy things that happen initially tend to smooth out later on.
- You have the star athlete who performs exceptionally well.
- There are all sorts of superstitions, jinxes, that surround these phenomena, so that star athlete doesn’t perform as well after being featured on the cover.
- They’re all sitting at their desks, and we’re going to give them an exam.
- As a result of taking that test, you’re going to have some people who took that test.
- Most people are going to get about 50 percent on average, but some people are going to perform better than 50 percent.
- With a group of about 100 people, I would imagine you’d have 2 or even 3 people who score 60 percent or even higher, say 65 percent.
- We put Billy over here, the top performer; Jane over here, she’s the bottom performer.
- Now what we want people to do is imagine what happens on day two.
- Day two, we perform exactly the same test, a different test, different questions, though it doesn’t really matter-they’re guessing anyway.
- What’s going to happen on day two? How would you expect Billy to perform on day two, and how would you expect Jane to perform on day two? We’ll ask the students to go ahead and make their predictions before we move on.
Episode 7 – Finding Things Out > The ebb and flow of random events > The ebb and flow of random events
- Now most people are going to appreciate that Jane and Billy in this case are going to perform on average on day two.
- Now what do we mean by that? Again, 100 people who took the exam in the first place-most of them are going to perform, by guessing alone, at 50 percent.
- Jane and Billy had a lump in their performance at 65 percent and 35 percent.
- So just like Jane who performed better the next time around-so she was at 35 percent; now she’s at 50 percent-you have the person who walked into a clinic at day one; they took a vial of water; and on day two, you test them again, and they improved, just by statistical chance alone.
- The 100 people can’t speak a word of English and, again, they take this exam.
- Now, on average, we expect people to perform at 50 percent again, Jane performs poorly on this exam and Billy performs well.
- Now before we separate Jane and Billy and we go on to the testing on day two, let’s do some interventions.
- Now we go to day two, and then they take the exam again.
- What do we expect? We expect Jane to regress towards the mean and do a little bit better on the exam, and we expect Billy, the second time, to drop back to regress towards the mean and do a little bit worse.
- Now, exactly like you said, if that happened to me, if I was Jane or Billy, I would attribute the change in their performance to the thing that I introduced.
- Most people who are taking this course might be quite willing just to dismiss people who fall into these tendencies as just being silly or stupid, but that’s not enough.
- I mean, if we can figure out why people tend to believe these things, I think we’re getting a lot further in figuring out when they’re operating and what we can do about them in that particular case.
- So just to make that concrete, let’s ask people at home just to answer a few more questions about regression to the mean before we move on.
Episode 7 – Finding Things Out > Small classes are better > Small classes are better
- We ask people, “Imagine yourself, given the choice between a large class, between taking a class with 500, 1,000 people sitting in this giant lecture hall watching the professor, or taking a small class, say 20 people, with a lecturer standing in front giving the same sort of thing, which would you prefer?” Now most people in that sort of context would say that they would prefer the small class as opposed to the large class.
- You’ll learn more-you’ll retain it for longer if you’re in a smaller class than in a larger class.
- There has been roughly half of the experiments coming out in favor of the small class, and roughly half of the experiments that have been done come out in favor of a large class, or no difference between them whatsoever.
- I want them to test the difference between a large class and a small class, design the experiment, and so they do.
- We’ve learnt a fair bit about the experimental method,” so they say things like, “Well, we have to control as much as possible, so you need the person at front of the class to be exactly-in front of a large class to be the same person in front of the small.
- ” Then I take another poll and say, “Okay, students, now how many of you are willing to change your mind? If I gave you the option of taking a large class and a small class now, how many people are going to take the small class and how many people are going to take the large class?” There’s no change.
- I think, going back to the large class-small class difference, for example, I would’ve been extremely surprised if I ask my students, after just presenting a couple of papers or on my authority, to say that, “No, there’s no difference between a large and a small class.
Episode 7 – Finding Things Out > Conversation with the MythBusters (Part 2) > Conversation with the MythBusters (Part 2)
- One of the things that you guys do extremely well, that I can’t say so much for my scientific colleagues, is admit when you’re wrong and recognize that we make mistakes.
- Does that happen often? Have you guys made many errors? Actually, that’s probably the most important question to ask here.
- It’s like the best possible thing that can happen to us-is to be wrong.
- The simple reason is that that’s an opportunity for us to learn something.
- When there’s a failure or if we’ve taken a particular course that has led us down a path that’s been unproductive, we have to stop and ask, “Well, why? What was it that I was doing wrong?” That’s when the light bulb goes off, and that’s when you’re on your road to do some new adventure or discovery.
- The two most important science teachers I had in all my schooling-both were willing, when I asked a question, to say, “Actually, that’s something I don’t know.” These are people in authority, and teachers generally don’t like to say they don’t know something.
- We did a story a bunch of years ago called “Killer Cable Snap.” Now there’s not a fisherman in the world who doesn’t believe that if a braided steel cable on a boat gets stretched to its breaking strength, that when it breaks, it can whip around and slice right through you.
- Now I’m eliminating aircraft carrier cables because those are about this big around, and that’s more like getting hit by a building.
- That’s not a “Whipping cable.” But there’s thousands of anecdotal reports, legs severed-and I’m also eliminating things like a boat rope pulling you against the dock because of the storm and slicing your legs off.
- We really thought we were going to cut these beautiful high-speed shots of the cable slicing right through.
- There are plenty of other ways to get sliced in half by ropes and cables on a boat, but whipping is not one of them, and we totally busted it, and we began the day totally believing we were going to totally confirm it.
- There can be some really stupid simple question about something that, “Of course that’s going to happen,” and then, “Darn, I didn’t think that wouldn’t happen.” In fact, given the vagaries of making a television show, there are times that we have to shoot out of order because we lose a location or we have some sort of snafu or something occurs.
- Occasionally, we have placed ourselves in a position of shooting out of order, going, “That’s okay. We can shoot out of order because we know what the results of that experiment are going to be.” We don’t do that anymore because every single time we’ve done it, every single time we’ve put our eggs in the basket of well-we-know-how-that’s-going-to-turn-out, we’ve been boned.
- The best known one was a plane on a conveyor belt, and the idea was that if a conveyor belt is going in one direction fast enough, a plane that’s sitting on it can’t go fast enough in the other direction and take off.
- That’s just… The question itself has a trick within it which leads you to believe that the plane will somehow remain stationary.
- Personally, I, on the other hand, I ask myself, “Is there anything that I actually simply believe in?” Belief sort of implies believing things or an understanding that does not require proof or evidence.
- Science is a religion that is based just as much on faith in science, as religion is based on faith in a supreme power, except that, as Tim Minchin says, “No, that’s total BS,” because science adjusts its views based on evidence.
- I think one of the strongest things that would actually say that we are scientists would be that both Adam and I have sort of a pact that if there is evidence to the contrary-like, in particular, when we’re arguing amongst each other over some particular thing that we’re trying to problem solve, as to how we problem solve it or what the results are-the minute that there’s clear evidence to the contrary, even though this may have been an idea that one of us has been championing just to the death that this is the way to do it, the minute that there’s anything that pops up on the radar that changes that, it’s point of pride for us to do immediate, without hesitation, back flip and agree with the other person or go off in a new direction.
- I would like to think that’s fundamental to science.
- When you’re on the veldt and you’re trying to kill an animal and you learn that one kind of sharpened stick works better at killing them than another, that’s a scientific experiment.
- You’ve taken variables; you’ve looked at them; you’ve compared them; and you’ve chosen the one that works better, really, that’s it.
Episode 7 – Finding Things Out > 6 leads of opinion change > 6 leads of opinion change
- The first of the six leads is for people to ask themselves: what do you really believe anyway? It’s really tempting to think that things are either for you or against you.
- The second of the six leads is: how well based is the opinion that you already hold? Now sometimes support for an opinion can’t always be based on formal data or experiments.
- Even if you do find a flaw, you have to ask yourself: is that flaw so terrible that you can legitimately disregard this new piece of evidence that you’ve been given? Is the data so bad that you can just ignore it? Like in the large class-small class example, are the experiments that have been done just so terrible that you can ignore them and continue to believe what you believe? That leads us to the fourth lead in opinion change, which is: does the evidence really contradict what you already believe? Is there a way of reframing the issue, of stating the evidence in a way that allows you to use this new information, this new evidence, rather than just rejecting it outright? Absolutely.
- Given the problem that’s in front of you, given the evidence that somebody’s presented, if that’s not enough for you to change your mind, change your opinion, then what would be enough? If I’ve just presented evidence or if I’ve just received evidence that’s somehow flawed or somehow insufficient, you need to ask yourself: if that’s not enough, what would be enough? Yes, and you need to-if you can’t find any evidence, if you can’t think of something that would change your mind, then it’s really time to be cautious there about the basis of that belief.
- I mean, yes we need a caveat on that sixth lead where people can really use that one almost as a get-out-of-jail-free card and just say, “Well, the cost is low, and therefore it really doesn’t require me to figure out what I actually believe about it.” So I’d be cautious with that one yes, if the cost is extremely low and the payoff is high, yes, why not? Just go with it.
- I think having these six leads is going to be really useful when people start examining their own opinions, and examining the opinions of other people.
- I’ve asked the MythBusters exactly about this issue of opinion change, both the opinions that they hold or the myths that they test, as well as other people’s, the viewer’s opinions that they have.
Episode 7 – Finding Things Out > The story so far > The story so far
- With the scientific method, this is a way that we need to deal with complex andambiguous events.
- Sometimes things are going to work for you, sometimes against, and you’regoing to improve sometimes after a lump of bad events, or drop after a bunchof good events, and this is just a natural process called regression to themean.
- When dealing with regression to the mean, this is just a statisticalphenomenon.
- As we’ve just seen, now we have six leads that we can use when people havestrong opinions about these kind of events.
- We’re going to pick up there next week when we deal, as I said, with somemore challenging types of events with extraordinary claims.
