How to make worked examples work

I’ve written a few times about how I use worked-out examples (and mistake analysis) in my teaching:

• Worked Examples and Loop-de-Loops
• Worked Examples and Geometry Proof
• Worked Examples and Focused Feedback
• Worked Examples and Feedback on a Whole Quiz

The resource I’ve returned to again and again is Algebra By Example, which is really well-done. (And get excited for Math by Example, their expansion pack for 4th and 5th Grade!)

I realize that there are two things that I haven’t yet shared: some research on how to design these things well, and my routine for using these materials in my teaching.

The Research

Whenever you get a bunch of people talking about research in education, somebody eventually mentions how sad it is that research in education usually happens without the input of educators. Wouldn’t it be grand if researchers and educators were more collaborative?

Yes! This indeed would be grand, and it should happen more often. But sometimes it does happen, and those partnerships can produce really wonderful things. Algebra by Example is the result of one of these partnerships.

The project was led by Julie Booth of Temple University, and Booth has written a very nice piece in an NCTM journal about the design process her team uses and the research behind it:

McGinn, K. M., Lange, K. E., & Booth, J. L. (2015). A worked example for creating worked examples. Mathematics Teaching in the Middle School, 21(1), 26-33.

Booth’s piece is very nice. It cites a more research-y piece that more carefully trawls and summarizes the worked example literature, and that is also an interesting (though more research-y) read:

Atkinson, R.K., et al. (2000). Learning from examples: Instructional principles from the worked examples research. Review of educational research, 70(2), 181-214.

Though it’s tough to summarize these pieces, here are a few important principles when designing example tasks:

1. Try to put everything in one place…
2. …but also try to visibly separate different steps.
3. Switch up correct examples with incorrect work.
4. Ask kids to explain what’s going on.
5. Ask kids to try it on their own.

Try to put everything in one place. That’s a NO for splitting the problem and the solution with a not-so-useful line of text. Keep words at a minimum; ideally, you can see the whole example at once.

Worse:

Better:

It’s useful (according to Booth’s research and also me) to switch up correct with incorrect examples.

Can you spot the error?

I like the way this incorrect work has some good thinking (I like the regrouping idea) and that it also clearly distinguishes each step.

From experience, I’ll say that people new to writing examples often try to do too much with the space they have. The goal really is to eliminate all but the most important thinking from the example, and it’s OK to use multiple examples to get at different aspects of the thinking. Overstuffing one of those little boxes can be a sign that it’s time to break this up into two smaller problems.

So much for the visual design of these materials, which is really a huge subject in of itself.

Some of the most interesting bits of those reviews are about self-explanation. Here’s a bit from the more research-y review:

Research on explanation effects suggests that self-explanations are an important learning activity during the study of worked examples. Unfortunately, the present research suggests that most learners self-explain in a passive or superficial way.

“Passive or superficial”: YES! All too often this is how students interact with an explanation (or with feedback).

So how do examples help at all, given the tendency of students to just passively say “OK fine yep I get it” when they see an example? I use a routine to really make example activities hum.

The Routine

Whenever I use an example activity with students, here is my routine.

1. Get ready: I show the problem, covering up everything besides the problem itself. I usually use a document camera, so I mean I literally cover it up with my hand. Sometimes I ask students to begin thinking about how they’d approach the problem before I reveal the student work.
2. Read: I ask everyone to read the student work to themselves. I ask students to offer a quiet thumb to let us know that they finished reading. I usually tell students to put a thumb up if they’ve read each line, even if they don’t understand everything fully yet.
3. Discuss: I assign partners (really I’ve already done this) and ask students to discuss the example until each person understands every line of the student work. (I focus my class on spreading understanding, not on solving problems.)
4. Explain: Every Algebra by Example problem has prompts for student explanation. When I make my own materials, I always include such prompts. It’s at this stage in the routine that I make sure everyone tries to answer the prompts. Then I usually call on a student to explain.
5. You Try: This is where we help students try it on their own. There’s a whole art to picking good problems for students to try here — they should change the surface-details in some way, while keeping the underlying ideas constant. This is where we try to keep kids from making false generalizations, and it’s another level of protection against superficial understanding.

Using the Algebra by Example materials, the routine flows from left to right:

It’s not so easy to design these prompts, either! Here are some suggestions, from Booth’s piece:

If you’re in the mood for a bit of practice, you might think about how you’d fill in the rest of this example task:

Example analysis works especially well as a feedback routine. There was a problem that kids especially had trouble with? Write an example that focuses on how to improve, and then follow the routine with some time to revise and improve the quizzes (or a re-quiz, or etc.).

All of this — the examples, the prompts, etc. — is worth your time, I think. It’s a lot of fun for kids to puzzle out how someone else is thinking, and the format allows us to really focus on a whole idea. It’s a bit of a myth, I think, that these types of problems are boring.

It’s also a myth that this kind of math is boring. It’s not, because there’s something to be understood here. And fundamentally, understanding is interesting.

Advertisement

Worked Examples and Loop-De-Loops

This is a Loop-de-Loop:

At camp, I taught a session on Loop-de-Loops, a mathematical object from Anna Weltman’s book. I had about 30 people in the room (15 counselors, 15 kids) looking for patterns and trying to figure out a bit about what Loop-de-Loops are all about.

I introduced Loop-de-Loops using Chris Lusto’s fantastic site, asking everyone what they noticed and were wondering after I hit “Show Me.” (Go try it; it’s a blast.)

This was perfect for us, as I wanted the focus of the session to be about asking questions. (Lord knows that the kids spend enough time solving problems in this camp. Asking questions is at least half of a full mathematical picture.) Lusto’s site makes it easy to quickly explore many different lists of numbers, generate theories and see if those theories hold up.

In fact, it’s too easy to generate these Loops with the software. For the sake of understanding, it’s always good to construct stuff by hand a couple times. Often we notice different things when we construct objects by hand than we do on the computer.

My first time running this session, though, I realized something: it’s hard to make a Loop-de-Loop!

Students and counselors struggled to draw these things — though, when they saw these objects on the computer, they were extremely confident that they understood how to make them. Surprise!

Today, I ran the session for another group of kids and counselors. This time, I came prepared. I wrote a worked example activity that aimed to help everyone better learn how to make these cool Loopy things:

I quickly made a handout right before class today. (Hence the marker.) I knew that I needed to include the three things I always include on one of these things:

1. The example, clearly distinguishing between the “task” and the “student work,” and trying to make sure to only include marks that contribute to understanding. (See those little arrow heads? I noticed that people had a hard time keeping track of directions while drawing these.) I tried to remove any distracting text or clutter — reading about the split attention effect helped me learn how to do this.
2. Prompts for noticing the most important stuff. Research suggests that students often don’t explain things to themselves, or do so superficially. (And experience totally confirms this.) Prompts, along with a clear call for students to spend a minute responding to the prompts, helps a lot with this I find.
3. A chance to try it out on your own, with the model nearby to help, if you get stuck.

I first read about this structure while reading about Cognitive Load Theory, but things didn’t click for me until I also saw the Algebra by Example project. Two other pieces have helped me better understand this bit of my teaching:

• Learning from Examples: Instructional Principles from the Worked Examples Research
• A Worked Example for Creating Worked Examples

Here’s how I do these things. First, I put the “student work” up on the board and ask everybody to silently study it. (People need time to think before talking!) Then, I ask everyone to check in with a neighbor and to take turns making sure each can explain what’s happening. (This is usually where there are “ah!”s and “oh wait!”s.) Then, we talk about the prompts. After that, I tell everyone to try it on their own.

This part of the session went so much better today than my first pass. Everyone was still challenged by drawing these, and there were still a lot of mistakes. But the difference between these two sessions was precisely the difference between productive and unproductive struggle. Instead of flailing around when they got lost in the construction, everyone had something to go back to. Ah, OK, so this is how it’s done.

Part of my job is also to help the counselors support the students in their math work. A lot of the counselors tell me they struggle with knowing about how much to give away to a student when they are stuck. And while I totally know what they mean, I always tell them that their job isn’t to give away stuff or to avoid giving stuff away. Their job is to get the ball rolling for the student, get thinking happening, as quickly as possible — and then to step back and let that thinking happen.

Today I needed to get the ball rolling. It was a session about posing problems, and I started with asking everyone to notice and ask questions. But an important part of getting the ball rolling was a worked example.

Study an example, see the world

I’ve been a math teacher in New York City since 2010, a few months after I graduated from college. It’s the only job I’ve ever had, besides for little things over the summer when I was a teen. (In order: babysitter, camp counselor, Pepsi vendor at Wrigley Field, tutor. All kind of relevant to teaching, come to think of it.)

Though I teach math, math didn’t feel easy for me as a student. It was never where I shined. An exception was geometry, with its heavy emphasis on proof. Proof felt natural for me in a way that algebra didn’t.

When I began teaching, I realized that for many students the situation is reversed — it’s proof that feels unnatural and cumbersome. Writing a proof involves combining statements in ways that seemed to mystify many students. This was especially true early in my career.

After a few years of hitting my head against the wall, I started to understand what made this such a difficult skill to teach. Proof is the closest that mathematics comes to writing, and writing itself is impossible without reading. How can a student who has never read an essay possibly write one? I concluded that my students needed to read more proofs.

It took me a few more years to understand how exactly to pull this off in class. My big frustration was that my students wouldn’t devote enough attention to the proof examples I shared. I would distribute a completed proof and ask the class to read it with care. Very often, it seemed that they missed the whole point of the proof. They couldn’t read it carefully yet — they didn’t know how.

Now, things go better when I share proofs in class. One big difference is I have a much better understanding of all the subtle conceptual understandings that go into a proof, many which were invisible to me at first. (In teaching, it can be trouble when a topic comes naturally to you.) There are many aspects of a proof that I need to help them uncover.

Besides for a better understanding of the subtleties of proof, I’ve learned to structure my activities in sturdier ways. I’ve learned to design these activities so that they have three parts:

• The proof example
• Comprehension questions about the example
• Proof-writing practice, with the example as a model

I didn’t come to this structure on my own, by the way. I came to it through reading about Cognitive Load Theory (where these are sometimes called “example-problem pairs”) and especially from seeing it in some especially well-designed curricular materials:

Also:

 

(In fact, I didn’t really understand how to make my own example activities until I saw many models in these curricular materials. I needed examples, myself.)

So, for instance, I created this proof example for my students this year:

Looking back, the example isn’t perfect. It ended up being a bit visually crowded, and it might have been better to eliminate some of the letter-abbreviations. In class, I actually covered up each stage of the proof to focus their attention on each part.

In any event, this activity shows a lot of what I’ve learned about teaching proof. I knew I wanted to make explicit the complicated two-stage structure of some congruence arguments, so I worked hard to create a pretty clear example for my students. I then called on students to answer a trio of analysis questions about the proof — there’s a lot to notice, and students don’t yet know how to notice the underlying structure of this kind of proof all on their own. Finally, I ask students to use what they’ve noticed on a related pair of problems, so that students see that there’s something here that’s generalizable to many different kinds of diagrams.

Even when my proof activities aren’t structured so rigidly, I try to include variety and a chance to practice. Here is a simpler activity, but I still call for students to do a bit of proof-completion in the second prompt:

Sometimes when I talk to other teachers about examples, they tell me they’re worried that kids will just try to unthinkingly copy the model. I do know what they mean, but it’s not what I see with my kids. I think that part of the reason is that I reserve example-analysis for when I worry that the math is going to be difficult, even overwhelming for many students. There is certainly a way to misuse these activities, and perhaps if I used these sorts of tasks on less complex material I would see unthinking imitation.

One of my jobs is to help students see things that they can’t yet see — things like the logical structure of a good mathematical argument, or the way just a tiny bit of information about a shape can guarantee a whole lot more. When things don’t come naturally to my students, what I’m learning to do is to design an activity that opens up a little window into the mathematics so they can learn to see new things.

Addendum (1/6/18): I just came across this lovely line from Paul Halmos:

A good stack of examples, as large as possible, is indispensable for a thorough understanding of any concept, and when I want to learn something new, I make it my first job to build one.

Teaching, in General

I. 

If you give a quiz covering lots of different topics, you’re going to get a lot of different mistakes. Which leaves you with a dilemma: how do you address those mistakes?

Yesterday’s quiz in geometry was a review quiz, so the topics were from all over the place:

• angles in isosceles triangles
• inscribed angles in a circle
• area of triangles, parallelograms and trapezoids
• congruence proofs

As expected, kids distributed their not-quite-there work fairly evenly across these topics. (OK so that’s not true, there were a lot of issues with the congruence proofs. There always are and always will be. Sigh.)

Here were two bad options for returning the quiz:

• Try to address all the issues with individual comments. First, it’s a game of whack-a-mole that is guaranteed to drive me insane. Second, what should I do? Try to leave perfect hints? Say nothing, and let kids figure out on their own what they did wrong? Show them the correct way to answer the question, and thereby eliminate anything for the kids to actually think about when I return the quizzes?
• Pick just one thing to focus on. Reteach that one thing in a careful way, then return the quizzes and ask kids to revise.

The second of the two options is great when there the mistakes are in the same galaxy. (I wrote about this in a post, Feedbackless Feedback.) But, I’m realizing now, this isn’t a terrific move when the mistakes are distributed across many topics. Because on what basis should I pick something to focus on reteaching? Any choice would be equally bad.

II. 

While reviewing the class’ quizzes, I found myself falling into written comments, at least until I figured out what else to do with the quizzes.

I used to write long, wordy comments that were essentially hints on the margins of the page. (“Great start! Have you tried multiplying both sides of the equation by 3?”) I came to dislike those sort of comments, as they just focus focus focus attention all on THIS problem. But I don’t particularly care about whether a students gets this problem correct; I care about the generalization.

What I’ve fallen into is, whenever possible, writing a quick example that’s related (but not identical) to the trouble-problem (the problem-problem) on the page. I do this below on the second question:

Then, I ask kids to revise the original on the basis of the example (or anything else they realized).

After writing a few of these example-comments, I realized I was taking a lot of time doing this, and repeating myself somewhat. I also realized that I don’t know if I could repeat this on every page for the congruence proofs, as the problem itself was reasonably complex:

Hard to read. I use the highlighter to flag errors, but this student highlighted the triangle himself.

I wasn’t sure what to do. Then, I remembered something I had read from Dylan Wiliam — I think it’s in Embedded Formative Assessment. His idea there was that you can give all the class’ comments to everyone, and then kids have to decide which comments apply to them.

I thought, OK, I can work with this. So I quickly (quickly!) made a page of examples, one for every mistake I saw on the quiz:

My routine in class went like this:

• Hand out the examples for revision.
• Hand back the quizzes with comments.
• Search for an example that’s relevant to your mistake.
• Call for revision on the basis of the examples. Work with friends, neighbors. Of course, I’m available to help.
• Then, try the extension task.

This was my first time trying this, but I thought it went well. Solid engagement, really good questions, no unproductively stuck students.

When you do something good in teaching, you never really know if it’ll work again, but I’ve got a good feeling about this one. It feels like a lot of what has already worked for me, but in a better order.

III.

Harry Fletcher-Wood is very nice and has a lot of interesting thoughts about feedback. As such, Harry and I very nicely disagree about a pretty interesting question about feedback: how can you teach people how to give better feeedback?

The usual caveats apply: I am not a teacher teacher, but Harry is involved in teacher education, and I have no idea if I’m right on this.

In any event, Harry recently published a really cool post where he tried to synthesize a lot of the research on feedback into a decision tree:

Now, this is awesome as a synthesis. But just because something is a good description of feedback doesn’t mean that it’s useful prescriptive advice. My favorite example of this comes from Pólya’s strategies for mathematical problem solving. Alan Schoenfeld has a nice way of putting it in Learning to Think Mathematically — the strategies have descriptive, but not prescriptive validity:

In short, the critique of the strategies listed in How to Solve It and its successors is that the characterizations of them were descriptive rather than prescriptive. That is, the characterizations allowed one to recognize the strategies when they were being used. However, Pólya’s characterizations did not provide the amount of detail that would enable people who were not already familiar with the strategies to be able to implement them.

In other words, just because a heuristic is a good description of practice doesn’t mean that it is an effective pedagogical tool. And that’s precisely my concern with Harry’s decision tree.

Feedback is a high-level concept that describes a TON of what happens in teaching. And any guidelines for how to give feedback effectively are also going to be high-level in a way that reminds me of Pólya’s moves like “find a simpler problem” or “draw a picture.”

And just as Pólya’s moves struggle because they aim to guide problem solving in geometry, algebra, topology, etc., all areas of math, Harry’s decision tree seems to me an attempt to guide feedback in all areas of teaching — math, history, medical school, etc.

Of course, Harry doesn’t intend for this to be the only thing guiding students, but neither did Pólya. My question is whether these generalizations themselves are helpful, beyond whatever ways that teacher educators can make them concrete and specific for teachers.

But what’s the alternative?

I don’t know yet. I can say a few things now that I couldn’t a few years ago:

• I think domain-specific — math-specific, history-specific — generalizations will be more useful than domain-general ones.
• I think that the generalizations can productively come in the form of instructional routines.

And, with this post and the other one, I now have two generalizations I can make about giving feedback in math class.

First: if there’s a problem that a lot of students have trouble with, consider a reteaching/revising cycle like the one in this image:

Second: if mistakes are sprinkled across too many topics, consider something like the revision routine I described in this post.

IV.

My bet is that a lot of knowledge about teaching looks like this. It’s not that there isn’t knowledge about teaching that accrues, but that we look for ways to scale things out of their contexts. Then we call those things myths and talk about how we have to kill ’em.

In general, generalizations about teaching are hard to come by. But nobody teaches in general. All teaching is intensely particular. These kids. These schools. This idea.

Some people are skeptical of the possibility of making generalizations about teaching, and the vast majority of people are cheery about making sky-high generalizations that cross every context. There’s a middle position that I want to find. There’s a sweet spot for knowledge about teaching, though I don’t know if we’ve all found it yet.

Feedbackless Feedback

I.

Not all my geometry students bombed the trig quiz. Some students knew exactly what they were doing:

A lot of my students, however, multiplied the tangent ratio by the height of their triangle:

In essence, it’s a corresponding parts mistake — the ’20’ corresponds to the ‘0.574’. The situation calls for division.

Half my class made this mistake on the quiz. What to do?

II.

Pretty much everyone agrees that feedback is important for learning, but pretty much nobody is sure what effective feedback looks like. Sure, you can find articles that promise 5 Research-Based Tips for great feedback, but there’s less there than meets the eye. You get guidelines like ‘be as specific as possible,’ which is the sort of goldilocks non-advice that education seems deeply committed to providing. Other advice is too vague to serve as anything but a gentle reminder of what we already know: ‘present feedback carefully,’ etc. You’ve heard this from me before.

As far as I can tell, this vagueness and confusion accurately reflects the state of research on feedback. The best, most current review of  feedback research (Valerie Schute’s) begins by observing that psychologists have been studying this stuff for over 50 years. And yet: “Within this large body of feedback research, there are many conflicting findings and no consistent pattern of results.”

Should feedback be immediate or delayed? Should you give lots of info, or not very much at all? Written or oral? Hints or explanations? If you’re hoping for guidance, you won’t find it here. (And let’s not forget that the vast majority of this research takes place in environments that are quite different from where we teach.)

Here’s how bad things are: Dylan Wilam, the guy who wrote the book on formative assessment, has suggested that the entire concept of feedback might be unhelpful in education.

It’s not looking like I’m going to get any clarity from research on what to do with this trig quiz.

III.

I’m usually the guy in the room who says that reductionist models are bad. I like messy models of reality. I get annoyed by overly-simplistic ideas about what science is or does. I don’t like simple models of teaching — it’s all about discovery — because I rarely find that things are simple. Messy, messy, (Messi!), messy.

Here’s the deal, though: a reductionist model of learning has been really clarifying for me.

The most helpful things I’ve read about feedback have been coldly reductive. Feedback doesn’t cause learning . Paying attention, thinking about new things — that leads to learning. Feedback either gets someone to think about something valuable, or it does nothing at all. (Meaning: it’s affecting either motivation or attention.)

Dylan Wiliam was helpful for me here too. He writes,

“If I had to reduce all of the research on feedback into one simple overarching idea, at least for academic subjects in school, it would be this: feedback should cause thinking.”

When is a reductive theory helpful, and when is it bad to reduce complexity? I wonder if reductive theories are maybe especially useful in teaching because the work has so much surface-level stuff to keep track of: the planning, the meetings, all those names. It’s hard to hold on to any sort of guideline during the flurry of a teaching day. Simple, powerful guidelines (heuristics?) might be especially useful to us.

Maybe, if the research on feedback was less of a random assortment of inconsistent results it would be possible to scrap together a non-reductive theory of it.

Anyway this is getting pretty far afield. What happened to those trig students?

IV. 

I’m a believer that the easiest way to understand why something is wrong is usually to understand why something else is right. (It’s another of the little overly-reductive theories I use in my teaching.)

The natural thing to do, I felt, would be to mark my students’ papers and offer some sort of explanation — written, verbal, whatever — about why what they did was incorrect, why they should have done 20/tan(30) rather than 20*tan(30). This seems to me the most feedbacky feedback possible.

But would that help kids learn how to accurately solve this problem? And would it get them to think about the difference between cases that call for each of these oh-so-similar calculations? I didn’t think it would.

So I didn’t bother marking their quizzes, at least right away. Instead I made a little example-based activity. I assigned the activity to my students in class the next day.

I’m not saying ‘here’s this great resource that you can use.’ This is an incredibly sloppy version of what I’m trying to describe — count the typos, if you can. And the explanation in my example is kind of…mushy. Could’ve been better.

What excites me is that this activity is replacing what was for me a far worse activity. Handing back these quizzes focuses their attention completely on what they did and what they could done to get the question right. There’s a time for that too, but this wasn’t a time for tinkering, it was a time for thinking about an important distinction between two different problem types. This activity focused attention (more or less) where it belonged.

So I think, for now, this is what feedback comes down to. Trying to figure out, as specifically as possible, what kids could learn, and then trying to figure out how to help them learn it.

It can be a whole-class activity; it can be an explanation; it can be practice; it can be an example; it can be a new lesson. It doesn’t need to be a comment. It doesn’t need to be personalized for every student. It just needs to do that one thing, the only thing feedback ever can do, which is help kids think about something.

The term ‘feedback’ comes with some unhelpful associations — comments, personalization, a conversation. It’s best, I think, to ignore these associations. Sometimes, it’s helpful to ignore complexity.