Feedbackless Feedback


Not all my geometry students bombed the trig quiz. Some students knew exactly what they were doing:

Screenshot 2017-05-26 at 3.12.57 PM

A lot of my students, however, multiplied the tangent ratio by the height of their triangle:

Screenshot 2017-05-26 at 3.19.05 PM.png

In essence, it’s a corresponding parts mistake — the ’20’ corresponds to the ‘0.574’. The situation calls for division.

Half my class made this mistake on the quiz. What to do?


Pretty much everyone agrees that feedback is important for learning, but pretty much nobody is sure what effective feedback looks like. Sure, you can find articles that promise 5 Research-Based Tips for great feedback, but there’s less there than meets the eye. You get guidelines like ‘be as specific as possible,’ which is the sort of goldilocks non-advice that education seems deeply committed to providing. Other advice is too vague to serve as anything but a gentle reminder of what we already know: ‘present feedback carefully,’ etc. You’ve heard this from me before.

As far as I can tell, this vagueness and confusion accurately reflects the state of research on feedback. The best, most current review of  feedback research (Valerie Schute’s) begins by observing that psychologists have been studying this stuff for over 50 years. And yet: “Within this large body of feedback research, there are many conflicting findings and no consistent pattern of results.”

Should feedback be immediate or delayed? Should you give lots of info, or not very much at all? Written or oral? Hints or explanations? If you’re hoping for guidance, you won’t find it here. (And let’s not forget that the vast majority of this research takes place in environments that are quite different from where we teach.)

Here’s how bad things are: Dylan Wilam, the guy who wrote the book on formative assessment, has suggested that the entire concept of feedback might be unhelpful in education.

It’s not looking like I’m going to get any clarity from research on what to do with this trig quiz.


I’m usually the guy in the room who says that reductionist models are bad. I like messy models of reality. I get annoyed by overly-simplistic ideas about what science is or does. I don’t like simple models of teaching — it’s all about discovery — because I rarely find that things are simple. Messy, messy, (Messi!), messy.

Here’s the deal, though: a reductionist model of learning has been really clarifying for me.

The most helpful things I’ve read about feedback have been coldly reductive. Feedback doesn’t cause learning . Paying attention, thinking about new things — that leads to learning. Feedback either gets someone to think about something valuable, or it does nothing at all. (Meaning: it’s affecting either motivation or attention.)

Dylan Wiliam was helpful for me here too. He writes,

“If I had to reduce all of the research on feedback into one simple overarching idea, at least for academic subjects in school, it would be this: feedback should cause thinking.”

When is a reductive theory helpful, and when is it bad to reduce complexity? I wonder if reductive theories are maybe especially useful in teaching because the work has so much surface-level stuff to keep track of: the planning, the meetings, all those names. It’s hard to hold on to any sort of guideline during the flurry of a teaching day. Simple, powerful guidelines (heuristics?) might be especially useful to us.

Maybe, if the research on feedback was less of a random assortment of inconsistent results it would be possible to scrap together a non-reductive theory of it.

Anyway this is getting pretty far afield. What happened to those trig students?


I’m a believer that the easiest way to understand why something is wrong is usually to understand why something else is right. (It’s another of the little overly-reductive theories I use in my teaching.)

The natural thing to do, I felt, would be to mark my students’ papers and offer some sort of explanation — written, verbal, whatever — about why what they did was incorrect, why they should have done 20/tan(30) rather than 20*tan(30). This seems to me the most feedbacky feedback possible.

But would that help kids learn how to accurately solve this problem? And would it get them to think about the difference between cases that call for each of these oh-so-similar calculations? I didn’t think it would.

So I didn’t bother marking their quizzes, at least right away. Instead I made a little example-based activity. I assigned the activity to my students in class the next day.


I’m not saying ‘here’s this great resource that you can use.’ This is an incredibly sloppy version of what I’m trying to describe — count the typos, if you can. And the explanation in my example is kind of…mushy. Could’ve been better.

What excites me is that this activity is replacing what was for me a far worse activity. Handing back these quizzes focuses their attention completely on what they did and what they could done to get the question right. There’s a time for that too, but this wasn’t a time for tinkering, it was a time for thinking about an important distinction between two different problem types. This activity focused attention (more or less) where it belonged.

So I think, for now, this is what feedback comes down to. Trying to figure out, as specifically as possible, what kids could learn, and then trying to figure out how to help them learn it.

It can be a whole-class activity; it can be an explanation; it can be practice; it can be an example; it can be a new lesson. It doesn’t need to be a comment. It doesn’t need to be personalized for every student. It just needs to do that one thing, the only thing feedback ever can do, which is help kids think about something.

The term ‘feedback’ comes with some unhelpful associations — comments, personalization, a conversation. It’s best, I think, to ignore these associations. Sometimes, it’s helpful to ignore complexity.


Reading Research: What Sort of Teaching Helps Struggling First Graders The Most?

I always get conflicted about reading an isolated study. I know I’m going to read it poorly. There will be lots of terms I don’t know; I won’t get the context of the results. I’m assured of misreading.

On the other side of the ledger, though, is curiosity, and the fun that comes from trying to puzzle these sort of things out. (The other carrot is insight. You never know when insight will hit.)

So, when I saw Heidi talk about this piece on twitter, I thought it would be fun to give it a closer read. It’s mathematically interesting, and much of it is obscure to me. Turns out that the piece is openly available, so you can play along at home. So, let’s take a closer look.


The stakes of this study are both high and crushingly low. Back in 2014 when this was published, the paper caught some press that picked up on its ‘Math Wars’ angle. For example, you have NPR‘s summary of the research:

Math teachers will often try to get creative with their lesson plans if their students are struggling to grasp concepts. But in “Which Instructional Practices Most Help First-Grade Students With and Without Mathematics Difficulties?” the researchers found that plain, old-fashioned practice and drills — directed by the teacher — were far more effective than “creative” methods such as music, math toys and student-directed learning.

Pushes all your teachery buttons, right?

But if the stakes seem high, the paper is also easy to disbelieve, if you don’t like the results.

Evidence about teaching comes in a lot of different forms. Sometimes, it comes from an experiment; y’all (randomly chosen people) try doing this, everyone else do that, and we see what happens. Other times we skip the ‘random’ part and find reasonable groups to compare (a ‘quasi-experiment‘). Still other times we don’t try for statistically valid comparisons between groups, and instead a team of researchers will look very, very closely at teaching in a methodologically rich and cautious way.

And sometimes we take a big pile of data and poke at it with a stick. That’s what the authors of this study set out to do.

I don’t mean to be dismissive of the paper. I’m writing about it because I think it’s worth writing about. But I also know that lots of us in education use research as a bludgeon. This leads to educators reading research with two questions in mind: (a) Can I bludgeon someone with this research? (b) How can I avoid getting bludgeoned by this research?

That’s why I’m taking pains to lower the stakes. This paper isn’t a crisis or a boon for anyone. It’s just the story of how a bunch of people analyzed a bunch of interesting data.

Freed of the responsibility of figuring out if this study threatens us or not, let’s muck around and see what we find.


The researchers lead off with a nifty bit of statistical work called factor analysis. It’s an analytical move that, as I read more about, I find both supremely cool and metaphysically questionable.

You might have heard of socioeconomic status. Socioeconomic status is supposed to explain a lot about the world we live in. But what is socioeconomic status?

You can’t directly measure someone’s socioeconomic status. It’s a latent variable, one responsible for a myriad other observable variables, such as parental income, occupational prestige, the number of books you lying around your parents’ house, and so on.

None of these observables, on their own, can explain much of the variance in student academic performance. If your parents have a lot of books at home, that’s just it: your parents have a lot of books. That doesn’t make you a measurably better student.

Here’s the way factor analysis works, in short. You get a long list of responses to a number of questions, or a long list of measurements. I don’t know, maybe there are 100 variables you’re looking at. And you wonder (or program a computer to wonder) whether these can be explained by some smaller set of latent variables. You see if some of your 100 variables tend to vary as a group, e.g. when income goes up by a bit, does educational attainment tend to rise too? You do this for all your variables, and hopefully you’re able to identify just a few latent variables that stand behind your big list. This makes the rest of your analysis a lot easier; much better to compare 3 variables than 100.

That’s what we do for socioeconomic status. That’s also what the authors of this paper do for instructional techniques teachers use with First Graders..

I’m new to all this, so please let me know if I’m messing any of this up, but it sure seems to me tough to figure out what exactly these latent variables are. One possibility is that all the little things that vary together — the parental income, the educational attainment, etc. — all contribute to academic outcomes, but just a little bit. Any one of them would be statistically irrelevant, but together, they have oomph.

This would be fine, I guess, but then why bother grouping them into some other latent variable? Wouldn’t we be better off saying that a bunch of little things can add up to something significant?

The other possibility is that socioeconomic status is some real, other thing, and all those other measurable variables are just pointing to this big, actual cause of academic success. What this ‘other thing’ actually is, though, remains up in the air.

(In searching for other people who worried about this, I came across a piece from History and Philosophy of Psychology Bulletin called ‘Four Queries About Factor Reality.’ Leading line: ‘When I first learned about factor analysis, there were four methodological questions that troubled me. They still do.’)

So, that’s the first piece of statistical wizardry in this paper. Keep reading: there’s more!


Back to First Graders. The authors of this paper didn’t collect this data; the Department of Education, through the National Center for Education Statistics, ran the survey.

The NCES study was immense. It’s longitudinal, so we’re following the same group of students over many years. I don’t really know the details, but they’re aiming for a nationally representative sample of participants in the study. We’re talking over ten-thousand students; their parents; thousands of teachers; they measured kids’ height, for crying out loud. It’s an awe-inspiring dataset, or at least it seems that way to me.

As part of the survey, they ask First Grade teachers to answer questions about their math teaching. First, 19 instructional activities…

Screenshot 2017-05-16 at 8.55.12 PM

…and then, 29 mathematical skills.

Screenshot 2017-05-16 at 8.55.48 PM

Now, we can start seeing the outlines of a research plan. Teachers tell you how they teach; we have info about how well these kids performed in math in Kindergarten and in First Grade; let’s find out how the teaching impacts the learning.

Sounds, good, except HOLY COW look at all these variables. 19 instructional techniques and 29 skills. That’s a lot of items.

I think you know what’s coming next…



So we do this factor analysis (beep bop boop boop) and it turns out that, yes, indeed some of the variables vary together, suggesting that there are some latent, unmeasured factors that we can study instead of all 48 of these items.

Some good news: the instructional techniques only got grouped with other instructional techniques, and skills got groups with skills. (It would be a bit weird if teachers who teach math through music focused more on place value, or something.)

I’m more interested in the instructional factors, so I’ll focus on the way these 19 instructional techniques got analytically grouped:

Screenshot 2017-05-16 at 9.08.53 PM.png

The factor loadings, as far as I understand, can be interpreted as correlation coefficients, i.e. higher means a tighter fit with the latent variable. (I don’t yet understand Cronbach’s Alpha or what it signifies. For me, that’ll have to wait.)

Some of these loadings seem pretty impressive. If a teacher says they frequently give worksheets, yeah, it sure seems like they also talk about frequently running routine drills. Ditto with ‘movement to learn math’ and ‘music to learn math.’

But here’s something I find interesting about all this. The factor analysis tells you what responses to this survey tended to vary together, and it helps you identify four groups of covarying instructional techniques. But — and this is the part I find so important — the RESEARCHERS DECIDE WHAT TO CALL THEM.

The first group of instructional techniques all focus on practicing solving problems: students practice on worksheets, or from textbooks, or drill, or do math on a chalkboard. The researchers name this latent variable ‘teacher-directed instruction.’

The second group of covarying techniques are: mixed ability group work, work on a problem with several solutions, solving a real life math problem, explaining stuff, and running peer tutoring activities. The researchers name this latent variable ‘student-centered instruction.’

I want to ask the same questions that I asked about socioeconomic status above. What is student-centered instruction? Is it just a little bit of group work, a little bit of real life math and peer tutoring, all mushed up and bundled together for convenience’s sake? Or is it some other thing, some style of instruction that these measurable variables are pointing us towards?

The researchers take pains to argue that it’s the latter. Student-centered activities, they say, ‘provide students with opportunities to be actively involved in the process of generating mathematical knowledge.’ That’s what they’re identifying with all these measurable things.

I’m unconvinced, though. We’re supposed to believe that these six techniques, though they vary together, are really a coherent style of teaching, in disguise. But there seems to me a gap between the techniques that teachers reported on and the style of teaching they describe as ‘student-centered.’ How do we know that these markers are indicators of that style?

Which leads me to think that they’re just six techniques that teachers often happen to use together. They go together, but I’m not sure the techniques stand for much more than what they are.

Eventually — I promise, we’re getting there — the researchers are going to find that teachers who emphasize the first set of activities help their weakest students more than teachers emphasizing the second set. And, eventually, NPR is going to pick up this study and run with it.

If the researchers decide to call the first group ‘individual math practice’ and the second ‘group work and problem solving’ then the headline news is “WEAKEST STUDENTS BENEFIT FROM INDIVIDUAL PRACTICE.” Instead, the researchers went for ‘teacher-directed’ and ‘student-centered’ and the headlines were “TEACHERS CODDLING CHILDREN; RUINING FUTURE.”

I’m not saying it’s the wrong choice. I’m saying it’s a choice.


Let’s skip to the end. Teacher-directed activities helped the weakest math students (MD = math difficulties) more than student-centered activities.

Screenshot 2017-05-16 at 9.39.26 PM.png

The researchers note that the effect sizes are small. Actually, they seem a bit embarrassed by this and argue that their results are conservative, and the real gains of teacher-directed instruction might be higher. Whatever. (Freddie deBoer reminds us that effect sizes in education tend to be modest, anyway. We can do less than we think we can.)

Also ineffective for learning to solve math problems: movement and music, calculating the answers instead of figuring them out, and ‘manipulatives.’ (The researchers call all of these ‘student-centered.’)

There’s one bit of cheating in the discussion, I think. The researchers found another interesting thing from the teacher survey data. When a teacher has a lot of students with math difficulty in a class, they are more likely to do activities involving calculators and with movement/music then they otherwise might be:

Screenshot 2017-05-16 at 9.48.00 PM

You might recall that these activities aren’t particularly effective math practice, and so they don’t lead to kids getting much better at solving problems.

By the time you get to the discussion of the results, though, here’s what they’re calling this: “the increasing reliance on non-teacher-directed instruction by first grade teachers when their classes include higher percentages of students with MD.”

Naming, man.

This got picked up by headlines, but I think the thing to check out is that the ‘student-directed’ category did not correlate with percentage of struggling math students in a class. That doesn’t sound to me like non-teacher-directed techniques get relied on when teachers have more weak math students in their classes.

The headline news for this study was “TEACHERS RELY ON INEFFECTIVE METHODS WHEN THE GOING GETS ROUGH.” But the headline probably should have been “KIDS DON’T LEARN TO ADD FROM USING CALCULATORS OR SINGING.”


Otherwise, though, I believe the results of this study pretty unambiguously.

Some people on Twitter worried about using a test with young children, but that doesn’t bother me so much. There are a lot of things that a well-designed test can’t measure that I care about, but it certainly measures some of the things I care about.

Big studies like this are not going to be subtle. You’re not going to get a picture into the most effective classrooms for struggling students. You’re not going to get details about what, precisely, it is that is ineffective about ineffective teaching. We’re not going to get nuance.

Then again, it’s not like education is a particularly nuanced place. There are plenty of people out there who take the stage to provide ridiculously simple slogans, and I think it’s helpful to take the slogans at their word.

Meaning: to the extent that your slogan is ‘fewer worksheets, more group work!’, that slogan is not supported by this evidence. Ditto with ‘less drill, more real life math!’

(I don’t have links to people providing these slogans, but that’s partly because scrolling through conference hashtags gives me indigestion.)

And, look, is it really so shocking that students with math difficulties benefit from classes that include proportionally more individual math practice?

No, or at least based on my experience it shouldn’t be. But the thing that the headlines get wrong is that this sort of teaching is anything simple. It’s hard to find the right sort of practice for students. It’s also hard to find classroom structures that give strong and struggling students valuable practice to work on at the same time. It’s hard to vary practice formats, hard to keep it interesting. Hard to make sure kids are making progress during practice. All of this is craft.

My takeaway from this study is that struggling students need more time to practice their skills. If you had to blindly choose a classroom that emphasized practice or real-life math for such a student, you might want to choose practice.

But I know from classroom teaching that there’s nothing simple about helping kids practice. It takes creativity, listening, and a lot of careful planning. Once we get past some of the idealistic sloganeering, I’m pretty sure most of us know this. So let’s talk about that: the ways we help kids practice their skills in ways that keep everybody in the room thinking, engaged, and that don’t make children feel stupid or that math hates them.

But as long as we trash-talk teacher-directed work and practice, I think we’ll need pieces like this as a correction.

Geometry Labs + Which One Doesn’t Belong

I love Henri Picciotto’s Geometry Labs text. I was preparing my geometry class for his inscribed angles activity, and saw this:


Thanks to the Which One Doesn’t Belong people (and Christopher’s lovely book), I’m no longer able to look at sets of four things. It’s ruined me. I’m always deciding which of them is the odd one out.

Since there are subtle differences between the inscribed angle cases, I decided to cover up the words and ask my students which of the four diagrams was the weird one.

image (10).jpg

This drew attention to the location of the centers, the location of radii, and the presence of isosceles/scalene triangles. (I know it’s May, but any chance to get kids to practice using this vocabulary is time well spent.)

This week in 4th Grade I’ve also been using Geometry Labs‘s chapter on tilings. (Sort of a random topic, but random topics are fun. Plus, I need to figure out where we stand on multiplication/division before one last push in our last weeks together.)

There I was, trying to figure out how to attune kids to the subtle classification differences between these two square tilings…


…and while, admittedly, I clearly had “Which One Doesn’t Belong” on my mind, it seemed a pretty good fit for my need here too. I took out some pattern blocks and snapped a picture:

image (9).jpg

There were lots of interesting aspects of this discussion, though my favorite had to do with whether the top-left and bottom-right tilings were different. I forget if we’ve talked about congruence yet in this class, but there were a lot of good ideas about whether tilting the tessellation made it a different tiling.

Not much else to share here, but I guess I’d say that I do this a lot. I don’t rewrite texts or worksheets or whatever very often. More often I add little activities before or after, to make sure kids can understand the activity, or to react to their thinking. That’s good for me (because I don’t have time to remake everything) and good for kids too (I write crappy curriculum).

What is it that I do?

I read a lot of teacher blogs these days.

(Incidentally, I turned MTBoS_Blogbot into an RSS feed, which was my reason for begging Lusto to make it in the first place.)

Anyway, I read a lot of teacher blogs. I see your beautiful activities, clever games and meaningful conversations. I wish I had an ounce of the teacherly creativity that Sarah Carter has, but really I don’t. It’s not what I do.

So, what exactly is it that I do?

In 8th Grade we’re going to study exponential functions. Class began with a lovely Desmos activity. They worked with randomly assigned partners.

Screenshot 2017-05-01 at 6.58.28 PM.png

After thinking through these questions, I thought kids could begin learning about equations for exponential functions, and towards this it would be helpful to contrast linear table/equations with exponential ones.

In years past, I would have aimed to elicit these ideas out of a conversation. I’ve lost faith in this move, though. While it’s nice to get kids to share ideas, their explanations are often muddy and don’t do much for kids who don’t already see the point. (Just because a kid can say something doesn’t mean that they should.) This, at least, is what I suspect.

Better, I’ve come to believe, to follow-up an activity like this one with briefly and directly presenting students with the new idea. I worry more about visual presentation than I used to. Here is what I planned to write on the board, from my planning notebook:


I put this on the board, so that it would be ready after the kids finished the Desmos activity: what could the equations of each of these relationships be? boom, here they are:

image (1).jpg
Spot the differences between this and my plan! They were all on purpose.

During planning I hadn’t fully thought through what I was going to ask kids to do with this visual. At first, I stumbled. I gave an explanation along with the visual, but I got vibes that kids weren’t really thinking carefully about the equations yet. So I asked them to talk to their partners for a minute to make sure they both understood where the exponential equation came from.

You can tell when a question like that falls flat. There wasn’t that pleasant hum of hard-thinking in the classroom, and the conversations I overheard were superficial.

Remembering the way Algebra by Example (via CLT) uses example/problem pairs, I quickly put a new question on the board. I posted an exponentially growing table and asked students to find an equation that could fit this relationship.

There we were! This question got that nice hum of thinking going.

The equation wasn’t there, originally, duh.

While eavesdropping on kids, I heard that L had a correct equation. I thought it would be good to ask L to present her response, as she isn’t one of the “regular customers.”

Her explanation, I thought, gave a great glimpse of how learning works. She shared her equation but immediately doubted it — she wasn’t sure if it worked for (0,5). After some encouragement from classmates she realized that it would work. Turns out that her thought process went like this: 10, 100, 1000, that’s powers of 10 and this looks a lot like that. But how can I get those 5s to show up in there too…ah! The example involved multiplication so this one can too.

(Of course, she didn’t say this in so many words. After class I complimented her on the explanation and she put herself down: I don’t know how to explain things. I told her that learning new stuff is like that — your mind outpaces your mouth — but I thought I had understood her, and confirmed that I got her process.)

With the example properly studied, I went on to another activity. Following my text, the next twist was to bring up compound interest. I worried, though, that my students would hardly understand the compound interest scenario well enough to learn something from attacking a particular problem.

While thinking about this during planning, I thought about Brian’s numberless word problems. (My understanding of numberless problems is, in turn, influenced by my understanding of goal-free problems in CLT.)

I took the example problem from my text ($600 investment, 7% interest/year, how much money do you have in 10 years?), erased the numbers and put the variables on the board.


Then, I asked kids (again with the partners) to come up with some numbers, and a question. If you come up with a question, try to answer it. (A kid asking But I can’t think of a question is why this activity was worth it. And with some more thought, they could.)

I collected their work from this numberless interest problem, and I have it in front of me now. I see some interesting things I didn’t catch during class. Like the kid who asked ‘How much $ does someone lose from interest after 5 years?’ (And why would an 8th Grader know what interest is, anyway?) Or the kids who thought a 10% interest rate would take $100 to $180 over 8 years.

No indications from this work that anyone uses multiplication by 1.10 or 1.08 or whatever to find interest. Not surprising, but I had forgotten that this would be a big deal for this group.

For a moment I’m tempted to give my class feedback on their work…but then I remember that I can also just design a short whole-group learning activity instead, so why bother with the written feedback at all.

I’m not exactly sure what ideas in the student work would be good to pick up on. I should probably advance their ability to use decimals to talk about percent increase, but then again there was also that kid who wasn’t sure what interest was.

My mind goes to mental math. I could create a string of problems that use the new, exponential structure with decimals:

  • 600 x 1.5
  • 600 x 1.5 x 1.5
  • 1000 x 1.5^3
  • 50% interest on a $200 investment

That’s awfully sloppy, but it’s just a first draft.

Or maybe the way to go is a Connecting Representations activity that asks kids to match exponential expressions with interest word problems.

I’m not sure, but all this is definitely a good example of what I do. It’s what I’m learning how to do better in teaching, at the moment. It’s not fancy or flashy, and no one’s lining up to give me 20k for it, but it’s definitely representative of where I am now.

I’m not sure at all how to generalize or describe what it is this is an example of, though. Is it the particular flow of the 45-minute session that I’m learning to manage? Or is it the particular activity structures that I happen to have gathered in my repertoire?

None of those are satisfying answers. Maybe, instead, this is just an example of me basically doing what I think I should be doing. My reading is piling up, and I’m getting some coherent ideas about how learning and teaching can work. This lesson is a good example of how those principles more-or-less look in action. It might not be right (and it sure isn’t at the upper limits of what math class can be) but I’ve got a decent reason for most of the decisions I made in this session.

I think what I have to share, then, is how what I’m reading connects to how I’m teaching. This episode is an example of that.