This result makes a lot of intuitive sense to me and I find it interesting that the researchers were so surprised. Something like it underlies a lot of folk-wisdom about learning ("practice makes perfect" etc.).
I agree with your final three points. Folk wisdom says that practice makes perfect but also recognizes that some people pick things up quicker than others. Maybe "learning" is identical, but "connecting the dots" isn't, or "generalizing from one example to another."
I'd be curious to see what the most important factor is.
Yeah, heebie's three points are important factors I think, and the link adds some useful nuance too.
The substance isn't my thing, but growth mixture models are what all the cool kids are asking for these days.
Obviously this is a lot easier to measure with math than with other subjects, but I think it does hold up with things like language immersion too.
It's interesting that it's so uniform. It's one thing to say "practice makes perfect", it's another thing to say "practice approaches perfection at a rate of between 15 and 18 percent per interaction."
I'm glad you feel vindicated by the spirit of the article. My first reaction was that it also vindicates automated learning processes, e.g. online homework platforms like this one, which seems like it would reduce human interaction in education.
> It's pretty easy to get students to follow an idea in isolation, but getting them to retain it and weave it into a larger picture is really difficult.
Yeah, I think one of the biggest difference between me and the smartest people I know is that they're often just correctly recalling some thing they sorted out carefully years ago and I'm here thinking "yeah, I knew that once, let's see if I can work it out again..." Note that this is only partly about pure recall, some of it is about better post-processing your ideas into forms that are more easily recallable.
The regular relationship they found was between learning and repetition of quality practice, so I suspect the difference might be how much quality practice different students get from the same amount of class time. You can read this a few ways:
- if they don't make any effort they won't learn (obviously)
- if they don't direct their effort efficiently* they won't get results commensurate with their effort
- if the teacher doesn't pay them any mind, they might not actually do the work
- if the teacher doesn't pay them any mind, they might not get good feedback
- if they don't deal with criticism well, they might not benefit from it
I suspect that part of being a good student, as well as just sweat, is both knowing what an opportunity to learn looks like off your own bat, through self-reflection, and being open to feedback that's usually going to be critical; making your own education. But the nice thing about this paper is that it suggests it's possible to design the course to provide these things.
*I can't remember any of the kids I grew up with or the students I knew who adopted the full Mr Gradgrind, chain the kid to a desk model who did really well. Lots and lots of low intensity time? I'll show you a B student at best.
"not time but opportunities"
I just don't believe this *at all*. Maybe it's true for super easy stuff, maybe it's true for people who haven't ever really learned how to learn, but thinking hard about something is super valuable.
Maybe there's some difference here where some people have learned how to go through multiple opportunity cycles with feedback on their own rather than needing a teacher to provide the other half of the process?
5: yup, language immersion is absolutely a case of this.
9: it wasn't "opportunities" in some broad sense, it was specific, structured exercises that were marked and returned with detailed feedback. In some sense the point might be consistent quality across the classroom.
I just don't believe this *at all*. Maybe it's true for super easy stuff, maybe it's true for people who haven't ever really learned how to learn, but thinking hard about something is super valuable.
I'm not sure what you mean by this. "Thinking hard" seems orthogonal to both time and number of opportunities to me.
Thinking hard takes time though!
Thinking about it more, what I need is an error theory. It is observable in every given domain that some people pick up skills quicker than others. Why is it that we all get this wrong? In the case of children and sugar, the explanation iirc was that sugar often goes along with exciting events (holidays), and so the children are hyper for lots of reasons that gets mistakenly attributed to the sugar.
But what's the explanation here? Some could be discounting early exposure, or miscounting "exposures.". But across all domains? No one picks up physical skills faster than anyone else? Are we counting exposure so broadly that we count transferrable skills as mini-exposures?
Like I think this is just for achieving a Chat-GPT level of ability to produce something that looks correct, not for true expertise.
Thinking hard takes time though!
Fair enough, so maybe not orthogonal to time. But surely thinking hard and trying to work through something multiple times is more effective than thinking hard for the same amount of time and only trying it once.
I also suspect that a lot of differences are the fault of people (and educational systems) tending to focus on the things that do seem to come easier. It is hard to be consistently mediocre, and being average doesn't get more resources or more fun.
Oh, hell, "opportunity" doesn't match "time.". So if we limit someone to eight math problems, they'll make the same relative rate of progress. But if we limit that group to an hour, some people are going to get eight problems, some are going to get sixty, and some are getting four. The sixty group gets more "exposure" because....they learned faster, if we take "faster" to have something to do with "time"?
15: huh? the observations were about *maths*
18 sounds reasonable, the other thing to keep in mind is that a big fraction give up after two repetitions.
Now you're getting it, 21 comments in from 7 unique commenters - pretty good, that's only three each on average.
is it too early to go ot? if so, apologies, but - heebie, i thought of you when i saw this small format apc briefcase for a v good price :
Ok, here's a question about the data: do they just throw out everyone who starts at their desired level of master from the beginning? On the one hand, it seems like they have to do that, there's lots of people who will just already know the material from a previous class (especially with Calculus where it's very normal to just take it twice). But on the other hand, it's essentially impossible to tell the difference between someone who gets most concepts on the first try and someone who already learned it in the past...
22: you get a pass because that's so beautiful.
20: if exposure isn't correlated to time, then describing it as "all differences in learning rates vanish" is more than a little misleading.
For example, my experience with Calculus 1 is:
Step 1: Take "Calculus the Easy Way" out of the library. Read it cover-to-cover several times slowly over the course of around two weeks. This was exhausting and I thought it was really hard. I did no exercises and had no homework and no teacher.
Step 2: Enroll in a college course on Calculus 1, expecting it to be difficult and to cover more material than what was in that book. Quickly discover that actually the book I'd read covers all of Calculus 1 and 2 (plus a tiny bit more), and get perfect scores on every assignment with no additional practice.
18, 25: Right, the finding is that learning rates are equal as a function of number of opportunities, not as a function of time.
When we have these kind of conversations I tell myself that my experiences trying to teach my stepdaughter arithmetic are so far outside the norm that they aren't relevant. I think that also might be the case for the experience described in 26.
My experience with Calculus* was that the teacher decided to trust us with the solutions in advance, so we got a booklet that had the solution to every homework problem for the whole year. The warning was that you'll probably do poorly on exams if you just copy the solutions. Towards the end of the fall, I did just copy the solutions for some stuff, which also happened to cover a type of problem where I had trouble with mistakenly reversing the signs** in a particular kind of calculation. I got a C on that exam.
Faced with the idea of actually doing the homework every night, something I found intolerable, I dedicated a week of winter break to doing as many homeworks as I could. I copied the problem numbers but not the solutions out of the booklet. Then I went through the book and did about 7-8 homeworks per day, getting almost to the very end of the course. For the next four months, I just turned in the homeworks I'd already written up during the winter break. I did fine, never got a C again.
I'm sure this is totally generalizable. After all, I had the same number of "exposures" as everyone else if exposure = homework.
*high school, but AP so more or less equivalent to college before colleges started making you take Calc again even if you did AP
**something something "washers" something? I remember learning in multivariable calculus a year later that there was an easier way.
Let's post our ages, birthdays, and number of calculus assignments completed.
Here's my theory, that is consistent with patterns I've noticed in Calculus grades, with what the data presented here, and with my own experience.
When a teacher explains new material to you, one of two things can happen:
1) You understand it conceptually and so can do the required exercises on the first try and even solve significant variations on the problem that weren't presented in class. You'll still make dumb arithmetic or copying errors now and then, but otherwise you'll get all the problems.
2) You don't understand it conceptually, and need repetition. After a certain number of repetitions you'll be able to get the questions right 80%+ of the time even though you don't really understand why you're doing what you're doing. If you're presented with a significant variation on the problem, or asked to solve a problem that involves combining what you learned with a different thing that you learned, then you're starting again from scratch and need repetitions again.
Note that it's very much possible for the same person to sometimes end up in category 1 and sometimes category 2 (e.g. based on how much focus they have that particular day, or how good the teacher is, etc.). Similarly, it's very common for people to first achieve a category 2 understanding and then only after that are they ready to understand the concepts.
Then the reason that in practice some people learn much faster than others is largely driven by what fraction of the time they end up category 1 vs. category 2, and not because they require less repetition when doing category 2 learning. Also some tasks (like learning vocabulary) are pure category 2 and cannot be done in a category 1 style.
My experience with Calculus I was that I sat in the back of the class behind a boy I was hopelessly in love with, and spent the hour drawing elaborate "tattoos" on his hands with multicolored ballpoint pens. At midterm progress reports I had an F, but I got an A for my final grade, so at some point I must have learned something, but I don't know what, or how. Then twenty-five years later heebie posted calculus tutorials online, which at first I enjoyed following, but then I got busy with something else and gave up. As of today, I do not know any calculus.
My experience with Calculus I was that I sat in the back of the class behind a boy I was hopelessly in love with, and spent the hour drawing elaborate "tattoos" on his hands with multicolored ballpoint pens
Do I understand correctly that he was holding his hand out for you to draw on? That doesn't sound like a hopeless situation.
At midterm progress reports I had an F, but I got an A for my final grade, so at some point I must have learned something, but I don't know what, or how
Did you fall out of love? Or did you run out of ideas for tattoos?
The tattoos were three dimensional surfaces and helped provide an understanding of how the surface area to volume ratio changed with movements of the boy's beautiful hand.
It probably depends on what you wrote.
Did everyone who speaks Spanish get together and come up with a new word for "refrigerator". Because I never heard of a "neveras".
On topic because of pedagogy.
And this was all happening while you were sitting *behind* him? How did you get at his hand? Did he just spend his lessons in a voluntary armlock?
The OP is fascinating and the commentary does, ironically, suggest that the nice friendly interactive classroom style might be a big reason for learning inequality: Who talks most in class? The students who are already doing well, who came in already knowing some of the material. Everytime I interact with them, I'm giving them more opportunities to learn. Their prior success breeds future success; they are more likely to speak up with me, with their peers, and they learn even faster. In contrast, a student who is behind is less likely to speak up and isn't getting that positive feedback. It naturally takes them longer to do the problems and thus they do fewer of them, spreading out their opportunities over a longer time. They do actually learn the material slower.
Definitely rings true.
But the problem is also perhaps that this is more fun for the teacher!
I think the same thing came up with the phonics debate IIRC?
Teachers are wedded to the provably ineffective "oh lets just talk to the kids about books and let them figure it out" approach rather than the effective "Cuh. Ah. Tuh. Cat." approach, partly because it's what they were taught works, but partly because it's much more enjoyable for them to be the Cool Teacher and engage informally with a few intelligent kids who are keen on reading, than to be the Cruel Teacher from "Another Brick in the Wall" and stand in front of twenty intimidated/bored children chanting Cuh Ah Tuh.
And if you do that with algebra, then no shit you'll spend most time talking and giving feedback to the bright kids who are getting it, because they will complete more exercises in the 45 minutes you've got with them, so there's more occasions for feedback, but also because they get it and so they will probably enjoy it, a bit, unlike the slower kids who are frustrated and ashamed and don't want to be there and aren't fun to talk to.
The machine doesn't have that problem. It will sit there patiently for hours and stare at the slow kid with unblinking and glacial patience until she works out that x=7, and then it'll recommend another exercise for her. It can't be bargained with, it can't be reasoned with. It doesn't feel boredom, or scorn, or impatience. And it absolutely will not stop until you have finished your worksheet.
32 I followed a different path to not knowing any calculus, but equally effective.
But the problem is also perhaps that this is more fun for the teacher!
Yes, this is a big problem. It's very easy for a bright-but-inconsiderate teacher to only teach to their A students. I think it's more of a problem at the college level than the elementary level, though. In elementary school, I think it's easy to break kids into groups according to reading level and sit down individually with each group, and focus on each student. At elementary school, there's really a sense that you're going to connect with every kid and light them up.
Middle school is when that notion dies. Partly because kids are wrestling with independence and maturity and what power do they have when someone tries to cajole them into cooperation, and may have good reason to be angry in their lives, but the only place they can express it without disproportionate consequences is the classroom. In conjunction with that, (and racism and bias and the rest of it) teachers are more willing to write off students as nonstarters. They start to only focus on the engaged kids. But I don't think they're exclusively focused on the A kids - they're focused on the stable, cooperative kids who participate in class.
I think the exclusive focus on the A kids really is a college problem. There's a basic presumption that we're not preoccupied by discipline issues, and there's a serious lack of training in pedagogy and being given huge classrooms that don't really lend themselves to individual connection anyway. So you have (some) inconsiderate teachers who are there to walk the A students through the material and blame everyone else for their own performance, and TAs and tutoring centers that pick up the slack.
42: I'd add that old school college pedagogy works to the extent that it does when everyone is a top student. When only 6% of the population attends, you're selecting for a group that could learn effectively from a dictionary or is rich enough that it doesn't matter.
But the other problem is that a lot of the opportunities are dependent on the motivation of the student. We can move a lot of the demands onto the professor, but if the student doesn't show up and doesn't do the work, for whatever reason, there's a limit to what one can do. Following good pedagogy just means that instead of failing the big stakes assignments, they fail all the small ones.
My wife is a primary school teacher in her first year of teaching* and it's remarkable, having watched her go through the training and qualification process, how much more she has been trained in when it comes to teaching pedagogy and how much it is grounded in research than I was ever taught when I was a university tutor/lecturer. The entire extent of my training for teaching was a couple of short workshops.
* in the UK you do a 2 year early career teaching program where you are already qualified and working professionally in schools but you have two years of additional mentoring and lesson observations and you have to document various things you are doing in terms of pedagogy, etc.
44: I had a similar experience - total of, I think, four days' training in teaching techniques and then I was fine to instruct on any non-specialist topic. Specialist subjects like firearms or driving needed a separate and rather more rigorous qualification before you can instruct, but anything where getting something wrong doesn't involve an immediate risk of violent death was fine.