There’s a lot of sloppy talk of science that gets tossed around education. Every teacher knows this — or least, I hope they do — and I try not to get too worked up about it. I also try not to get worked up by people who wear backpacks on crowded subways. Not that it’s OK, but lots of people do it (the backpack thing) and picking any one person to bark at hardly seems like it would help.
The other thing is that people can be really passionate about sloppy science. This happens all the time, it’s nothing special about education. But passion makes it hard to talk critically about the research without it seeming like you’re attacking everything else that the person is passionate about.
I think a lot of the time it’s because we assume that the attack on the research isn’t really about the research, it’s about everything else it supports. It’s like, Why are you bothering to poke holes in [research that supports X]? You’d only do that if you were really against [X].
Which leads to an interesting question. Is it possible, at all, to avoid this trap? Is it possible to critique sloppy use of research without being heard as if you’re trashing a person, their organization, and everything they stand for?
What if you ask a lot of rhetorical questions — does that help?
Anyway, let’s talk about YouCubed. I think there’s something that — if we slow down, turn off passion, turn on curiosity — we can agree is a serious mistake. Here’s a popular quote from their popular page, Mistakes Grow Your Brain:
When I have told teachers that mistakes cause your brain to spark and grow, they have said, “Surely this only happens if students correct their mistake and go on to solve the problem correctly.” But this is not the case. In fact, Moser’s study shows us that we don’t even have to be aware we have made a mistake for brain sparks to occur.
When teachers ask me how this can be possible, I tell them that the best thinking we have on this now is that the brain sparks and grows when we make a mistake, even if we are not aware of it, because it is a time of struggle; the brain is challenged and the challenge results in growth.
Indeed — this does sound really, really surprising! So, applied to math, it sounds like if you solve an equation incorrectly you learn from that experience (brain sparks?) in a way that you wouldn’t if you had solved it correctly. The mistake you made causes struggle — even if it’s not a struggle that you’re aware of. You might not feel as if you’re struggling, but your brain is on account of the mistake.
The citation of Moser is very clear, so, ok, let’s go upstream and check out Moser. Though a lot of research is behind paywalls, a lot of it isn’t and a quick Google Scholar search gives us a copy of the paper, here.
The study was an
fMRI (EEG, see below. -MP) study. Here’s my understanding of the paper. The researchers wanted to know, on a neurological level, what’s different about how people with a growth mindset or a fixed mindset react to mistakes. So they gave everybody a test, to figure out if they had a fixed or a growth mindset. Then they hooked subjects up to fMRI EEG machines. While in the machines, participants worked on a task that yields a lot of mistakes. Researchers recorded their neural activity and then analyzed it, to see if there was a meaningful difference between the fixed and growth mindset groups, after making errors.
Spoilers: they found a difference.
Also: the whole paper is premised on people being aware of the errors that they made. This is not a subtle point, buried in analysis — the paper mentions this like nine times, including towards the end where they write, “one reason why a growth mind-set leads to an increased likelihood of learning from mistakes is enhanced on-line error awareness.”
This is so clearly different than what the YouCubed site says that I’m starting to doubt myself. The paper seems to be entirely about what happens when you realize you’ve made a mistake. Yet it’s cited as supporting the notion that you learn (“brain grows”) from mistakes, even if you aren’t aware of them.
Is there something that I’m missing here?
This doesn’t seem to me like an isolated issue with YouCubed.
A while ago Yana Weinstein and I made a document together trying to collect errors in YouCubed materials, explain why, and suggest improvements. I don’t have much else to say about that, except that if you’re interested you might check it out here.
What strikes me about YouCubed is that the errors just seem so unnecessary. The message is a familiar one, and I’m OK with a lot of it: don’t obsess over speed, think about mindset, don’t be afraid of mistakes. But there’s this sloppy science that gets duct taped on to the message. What purpose does that serve?
There’s also the question of why so few people in the math education community talk about this. I mean, it’s not like we lack the critical capacity. Every so often I see people in math education whip out their skeptical tools to tear apart a piece of research. Why not with YouCubed?
I don’t want to be cynical, but I want to be truthful. The first reason, I think, is because the message of YouCubed (besides the science) is widely admired. A lot of teachers love it, and nobody wants to ruin a fun time.
But I don’t think we have to worry about that. We can talk about the science of YouCubed in a way that doesn’t entangle the rest of the YouCubed message.
The second reason is, I think, that YouCubed and Jo Boaler’s popularity makes it difficult for the most visible people in math education to seem critical — no one wants to turn on one of their own. Especially since Boaler has often been subject to unreasonable attacks in the past — nobody wants to be unfair, cruel or sexist to her.
I’m actually very sympathetic to that. But this is also why it’s important for people who aren’t part of the research or PD world to have platforms to discuss ideas. We don’t have the reputation or the connections to lose, and so we can take a closer look and ask, wait, does that really make sense?
Actual neuroscientist Daniel Ansari points out that I don’t know the difference between an EEG and an fMRI, which is true.
13 thoughts on “YouCubed is Sloppy About Research”
Thank you for this. I believe Jo Boaler is on the right track and is generally an excellent evangelist for methods that improve math education, but the sloppiness bothers me too. For instance, when I was doing some research and documentation around formative assessment, I wanted to look into a result she frequently cites about written comments. Eventually I tracked it down as Geoff Cohen et al., Journal of Experimental Psychology: General 2014, Vol. 143, No. 2, 804 – 824, and I summarized it this way:
“I’m giving you these comments because I have very high expectations and I know that you can reach them” yielded much better effort & performance (especially for African American students) than “I’m giving you these comments so that you’ll have feedback on your paper.”
Especially when supported by the graphs & data in the paper, that’s a really useful and impressive result. But for some reason Jo Boaler describes it in her MOOC and elsewhere as “I am giving you this feedback because I believe in you.” I mean, it’s not that it’s that different, or that I think it’s a sign she’s cheating… but it’s sloppy, and the sloppiness undermines her point and gives her critics ammunition.
I have similar issues with Peter Liljedahl — I love love love what his ideas have done for my teaching, but when I tried a couple of years ago to look into what his research results actually were, there were lots of citations pointing to unpublished results. In fairness, maybe some of those have been published since then, but it did undermine my enthusiasm at the time, and it makes me more skeptical than I would otherwise be of some of his latest writing and ideas.
Also, thank you for (in your linked document) calling out the “everyone can learn maths to the highest levels” statement. It always creeps me out because we all KNOW it’s not true and I feel like it’s a refusal to acknowledge severely disabled learners as people.
LikeLiked by 2 people
I love your questioning, as always. The mistakes claim (repeated in Mathematical Mindsets) always bugged me, but I never read upstream. I think that it’s respectful of Boaler’s work to ask these questions, as well as circumventing outside unsympathetic criticism. Not having been subject to these questions is a valid criticism.
LikeLiked by 1 person
In the social sciences it’s not uncommon to cite results that for one reason or another are not published. In linguistics, for example, people often cite unpublished results that are older than they are!
Interesting. I guess I’m implicitly comparing against chemistry, which is the field I was originally trained in for reading research papers. The rules (both written and unwritten) seem a lot more rigorous in that field.
Thank you for this – I’m reading her book at the moment and doing the Stanford course. I keep getting distracted by my need to try and dig up the research that some of her claims are based upon after reading the Moser paper and failing to see how it supported her conclusions on errors.
LikeLiked by 1 person
Thanks for a great post it would be great if more teachers checked the research as that is what science is about – peer review.
I’ve done a similar thing for Hattie’s work and found the same problems – misinterpretation and mistakes – for example, Hattie’s class size research see here – https://visablelearning.blogspot.com.au/p/class-size.html
Also, has anyone seen some good critiques of Marzano?
LikeLiked by 1 person
I get that we need to be critical of things and challenge research but, to answer your question, it does come across as trashing their work when:…
I) YouCubed are trying to use research to dispel huge and serious myths that restrict learning, so we can pick holes but as far as I can see all of your corrections say pretty much the same thing in terms of content.
II) Your corrections seem basically a really nice job of copywriting to ensure the meaning of every word is spot on. Great, share this with the YouCubed team, don’t bash their integrity in terms of research and science. Maybe ‘Sloppy Grammar and Spelling’ is a more appropriate title for this post?
III) Sloppy Science? This isn’t fake news here. The YouCubed team cite all the research and encourage people to check out that what they are saying is backed up. They actually encourage educational discourse and not cherry picking soundbites to misrepresent findings.
IV) Your critique of the language also seems unfair when the website you get this from is a student/parent/teacher friendly one that encourages interest in Maths. Seeing as you love a question…which message will a kid who believes they are stupid and can’t do Maths respond better to, “Nearly everyone can do Maths” or “EVERYONE can do Maths”?
LikeLiked by 1 person
Re the document critiquing their language and word/choice, here is what Jo Boaler said:
But of course most of this piece is about an interpretation of research that seems more significant and more significantly incorrect. Do mistakes grow your brain, even if you aren’t aware of them? Citing Moser, YouCubed says “yes,” but the cited paper doesn’t say that. That’s the most important aspect of this.
Just wanted to say I read the document with the suggestions for correcting certain statements made on YouCubed and was very impressed. I understand that people might not want to be critical of someone so popular, but she is deeply involved in creating/influencing the math curriculum in the U.S.
LikeLiked by 1 person
I teach in SFUSD, where J. Boaler’s ideas have taken hold in force. The result: I’ve seen a decrease in mathematics knowledge acquisition over a decade. To say that the mathematics curriculum has been “dumbed down” is not too far off the mark. Boaler’s ideas do not work. Her published research does not hold up under scrutiny. While a passionate and excellent speaker, her scholarship is sorely lacking. That’s how she influences people: she uses emotions and taps into “math anxiety,” “math phobia,” and, well, hatred of mathematics, to make her arguments sound like common sense.
All it takes is time to read through her citations. She often cites herself or close ideological allies. When you read those papers, it’s all echo chamber. And often they are not valid, controlled “gold standard” research papers. They’re opinion pieces masquerading as research.
I think we’re in a dark ages for research in America. Our academic institutions are being compromised by sloppy methodology, criticism is being stifled by fear of being “critical,” and ideology and bias are trumping facts.
Tough time to be a math teacher in America.
LikeLiked by 1 person
Proof by Ted talk, English accent, and amazing charm. What more do you need? We elect our leaders on less.
LikeLiked by 1 person