YouCubed is Sloppy About Research

I.

There’s a lot of sloppy talk of science that gets tossed around education. Every teacher knows this — or least, I hope they do — and I try not to get too worked up about it. I also try not to get worked up by people who wear backpacks on crowded subways. Not that it’s OK, but lots of people do it (the backpack thing) and picking any one person to bark at hardly seems like it would help.

The other thing is that people can be really passionate about sloppy science. This happens all the time, it’s nothing special about education. But passion makes it hard to talk critically about the research without it seeming like you’re attacking everything else that the person is passionate about.

I think a lot of the time it’s because we assume that the attack on the research isn’t really about the research, it’s about everything else it supports. It’s like, Why are you bothering to poke holes in [research that supports X]? You’d only do that if you were really against [X]. 

Which leads to an interesting question. Is it possible, at all, to avoid this trap? Is it possible to critique sloppy use of research without being heard as if you’re trashing a person, their organization, and everything they stand for?

II.

What if you ask a lot of rhetorical questions — does that help?

Anyway, let’s talk about YouCubed. I think there’s something that — if we slow down, turn off passion, turn on curiosity — we can agree is a serious mistake. Here’s a popular quote from their popular page, Mistakes Grow Your Brain:

When I have told teachers that mistakes cause your brain to spark and grow, they have said, “Surely this only happens if students correct their mistake and go on to solve the problem correctly.” But this is not the case. In fact, Moser’s study shows us that we don’t even have to be aware we have made a mistake for brain sparks to occur.

When teachers ask me how this can be possible, I tell them that the best thinking we have on this now is that the brain sparks and grows when we make a mistake, even if we are not aware of it, because it is a time of struggle; the brain is challenged and the challenge results in growth.

Indeed — this does sound really, really surprising! So, applied to math, it sounds like if you solve an equation incorrectly you learn from that experience (brain sparks?) in a way that you wouldn’t if you had solved it correctly. The mistake you made causes struggle — even if it’s not a struggle that you’re aware of. You might not feel as if you’re struggling, but your brain is on account of the mistake.

The citation of Moser is very clear, so, ok, let’s go upstream and check out Moser. Though a lot of research is behind paywalls, a lot of it isn’t and a quick Google Scholar search gives us a copy of the paper, here.

The study was an fMRI (EEG, see below. -MP) study. Here’s my understanding of the paper. The researchers wanted to know, on a neurological level, what’s different about how people with a growth mindset or a fixed mindset react to mistakes. So they gave everybody a test, to figure out if they had a fixed or a growth mindset. Then they hooked subjects up to fMRI EEG machines. While in the machines, participants worked on a task that yields a lot of mistakes. Researchers recorded their neural activity and then analyzed it, to see if there was a meaningful difference between the fixed and growth mindset groups, after making errors.

Spoilers: they found a difference.

Also: the whole paper is premised on people being aware of the errors that they made. This is not a subtle point, buried in analysis — the paper mentions this like nine times, including towards the end where they write, “one reason why a growth mind-set leads to an increased likelihood of learning from mistakes is enhanced on-line error awareness.”

This is so clearly different than what the YouCubed site says that I’m starting to doubt myself. The paper seems to be entirely about what happens when you realize you’ve made a mistake. Yet it’s cited as supporting the notion that you learn (“brain grows”) from mistakes, even if you aren’t aware of them.

Is there something that I’m missing here?

III. 

This doesn’t seem to me like an isolated issue with YouCubed.

A while ago Yana Weinstein and I made a document together trying to collect errors in YouCubed materials, explain why, and suggest improvements. I don’t have much else to say about that, except that if you’re interested you might check it out here.

IV.

What strikes me about YouCubed is that the errors just seem so unnecessary. The message is a familiar one, and I’m OK with a lot of it: don’t obsess over speed, think about mindset, don’t be afraid of mistakes. But there’s this sloppy science that gets duct taped on to the message. What purpose does that serve?

There’s also the question of why so few people in the math education community talk about this. I mean, it’s not like we lack the critical capacity. Every so often I see people in math education whip out their skeptical tools to tear apart a piece of research. Why not with YouCubed?

I don’t want to be cynical, but I want to be truthful. The first reason, I think, is because the message of YouCubed (besides the science) is widely admired. A lot of teachers love it, and nobody wants to ruin a fun time.

But I don’t think we have to worry about that. We can talk about the science of YouCubed in a way that doesn’t entangle the rest of the YouCubed message.

The second reason is, I think, that YouCubed and Jo Boaler’s popularity makes it difficult for the most visible people in math education to seem critical — no one wants to turn on one of their own. Especially since Boaler has often been subject to unreasonable attacks in the past — nobody wants to be unfair, cruel or sexist to her.

I’m actually very sympathetic to that. But this is also why it’s important for people who aren’t part of the research or PD world to have platforms to discuss ideas. We don’t have the reputation or the connections to lose, and so we can take a closer look and ask, wait, does that really make sense?

Corrections:

Actual neuroscientist Daniel Ansari points out that I don’t know the difference between an EEG and an fMRI, which is true.

 

Advertisements

6 thoughts on “YouCubed is Sloppy About Research

  1. Thank you for this. I believe Jo Boaler is on the right track and is generally an excellent evangelist for methods that improve math education, but the sloppiness bothers me too. For instance, when I was doing some research and documentation around formative assessment, I wanted to look into a result she frequently cites about written comments. Eventually I tracked it down as Geoff Cohen et al., Journal of Experimental Psychology: General 2014, Vol. 143, No. 2, 804 – 824, and I summarized it this way:

    “I’m giving you these comments because I have very high expectations and I know that you can reach them” yielded much better effort & performance (especially for African American students) than “I’m giving you these comments so that you’ll have feedback on your paper.”

    Especially when supported by the graphs & data in the paper, that’s a really useful and impressive result. But for some reason Jo Boaler describes it in her MOOC and elsewhere as “I am giving you this feedback because I believe in you.” I mean, it’s not that it’s that different, or that I think it’s a sign she’s cheating… but it’s sloppy, and the sloppiness undermines her point and gives her critics ammunition.

    I have similar issues with Peter Liljedahl — I love love love what his ideas have done for my teaching, but when I tried a couple of years ago to look into what his research results actually were, there were lots of citations pointing to unpublished results. In fairness, maybe some of those have been published since then, but it did undermine my enthusiasm at the time, and it makes me more skeptical than I would otherwise be of some of his latest writing and ideas.

    Also, thank you for (in your linked document) calling out the “everyone can learn maths to the highest levels” statement. It always creeps me out because we all KNOW it’s not true and I feel like it’s a refusal to acknowledge severely disabled learners as people.

    Liked by 2 people

  2. I love your questioning, as always. The mistakes claim (repeated in Mathematical Mindsets) always bugged me, but I never read upstream. I think that it’s respectful of Boaler’s work to ask these questions, as well as circumventing outside unsympathetic criticism. Not having been subject to these questions is a valid criticism.

    Liked by 1 person

  3. In the social sciences it’s not uncommon to cite results that for one reason or another are not published. In linguistics, for example, people often cite unpublished results that are older than they are!

    Like

    1. Interesting. I guess I’m implicitly comparing against chemistry, which is the field I was originally trained in for reading research papers. The rules (both written and unwritten) seem a lot more rigorous in that field.

      Like

  4. Thank you for this – I’m reading her book at the moment and doing the Stanford course. I keep getting distracted by my need to try and dig up the research that some of her claims are based upon after reading the Moser paper and failing to see how it supported her conclusions on errors.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s