High-stakes for whom?
Let’s talk about so-called high-stakes testing. We’re all familiar with the term, and most of us are vaguely familiar with at least some of the tests. The English and Math tests administered to all students in grades 3-8 are of particular interest to New Yorkers. They’re the focus of most of our instruction for most of the year, and the subject of innumerable worried articles in our local newspapers. They sparked the opt-out movement on Long Island and in other middle-class areas and have been the focus of much wrangling between the city, the DOE, and the teacher’s union. Despite the arguing, the tests are administered every year. Huge amounts of time and money are devoted to preparing for the tests, administering, scoring them, and creating detailed statistical analyses of the results.
So, what’s the deal? Why are tests administered to school children worth such angst and arguing? Surely the results of these tests must be terribly important, if they get people so worked up.
People who are directly involved in the NYC school system would probably be surprised to learn what is done with the lovingly-tabulated score reports, charts, graphs, and spreadsheets that track student scores on the individual, school, and citywide level, even disaggregating the data by race, gender, native language, and disability status.
Practically nothing is done with them, and what is done is surprising.
Test scores cannot be used as the primary basis for promotional decisions. To translate that out of edu-speak and into the form of English used by actual humans—your kid doesn’t have to pass the test to move on to the next grade. Although test scores could, theoretically, be a part of promotional decisions, they aren’t. Your kid will go on to the next grade if got passing scores on his report card. Even if he didn’t, he’ll probably still move on (more on that another time). Test scores may, sometimes, be used to make placement decisions for entering ninth-graders, but that’s about it. That’s the only real ‘stake’ the kids have in tests—so I don’t think we can say that they are high-stakes for them. Any parent or teacher will tell you that using the tests as promotional criteria would be wildly unjust because they are not an accurate or sufficient measure of what students have learned.
Contrary to popular belief, the state tests scores aren’t used to rate teachers either. We do get an end-of-year rating, calculated through a Byzantine process that does include a totally different set of tests (more on that later, too) but the Common Core tests that make the news do not make up any part of our rating. That means those scores aren’t and can’t be used for hiring, firing, promoting, etc. You’d never know it from the way teachers tend to behave, but we don’t really have much of a stake in those tests. Despite all the hours we devote to them and the lectures we give our students about taking them seriously, they have no individual consequences for us at all. Any teacher or administrator, and most higher-ranking education officers, will tell you that using test scores for employment decisions would be wildly unjust because they are not an accurate or complete measure of what happens in a given teachers’ classroom.
All that data is used for something, though. It’s used to rate and make decisions about schools. It’s a complex and virtually impenetrable system, but state tests scores are one key component in deciding whether or not a given school should receive increased oversight, go into receivership, or (eventually) be shut down. Of course, that’s not going to happen to very many schools. For the majority, their scores will just be something to brag about or to bemoan. For some schools, however, those scores are going to make a huge difference. In some schools, those scores are going to mean massive restructuring, increased workloads, increased oversight, lost jobs, and ruined careers. You won’t hear much complaining about this. You will hear a lot of people agreeing that underperforming schools need to change or be closed, that equity requires dramatic action to ensure that all students have ‘access’ to a quality education.
What’s going on?
It seems to me that this use of data is wildly illogical and internally contradictory. It seems to be the state’s (and the city’s) policy that test scores are not reliable enough to make the relatively minor decision of whether or not an individual student should go on to the next grade –but are reliable enough to make the major decision to shut down a whole school. They not reliable enough to make the comparatively low-stakes decision to fire (or promote!) any individual teacher—but are reliable enough to make the much larger decision to fire a whole school’s staff and force them to reapply for their jobs. How does that work? If test scores can’t give you an accurate picture of learning for one student, or one classroom, how can they give you an accurate picture of learning for a school?
Can anyone explain this to me? Can anyone explain why test results are an unfair means of judging a student or a teacher, but a fair means of judging a school?
It seems to me that it ought to work in the opposite direction—I can easily imagine a situation where an average school has a few teachers with noticeably higher test scores; it seems to me that such data could tell you which teachers are doing better than you would expect, given the environment. I can imagine a high-performing school with a few classes of lower scores; such data might tell you which teachers are doing worse than you would expect, given the environment. If we consider students, the same general idea applies. If you have a class with low-to-average scores and some students perform well, you might reasonable conclude that they have learned more than their peers in the same environment. If you have a high-scoring class and some students with poor scores, you could reasonably conclude that those students do not measure up to their environment. I cannot see how it works the other way. If the scores of my students in my class don’t tell you anything meaningful about my performance as a teacher, then how can the average scores of the whole school tell you anything meaningful about me or my students?
It makes no sense. Either the test scores are reliable and important data, in which case they can and should be used to make both individual and school-level decisions, or they are not, in which case they should not be used to make any decisions.
So, what’s up? What kind of game is the DOE playing? I have a few ideas.
Let’s take students first. In 2018, 46.7% of NYC students were proficient in ELA. 42.7% were proficient in math. If these tests were used to make promotional decisions (as they were, once, meant to do) then more than half of the city’s kids would be left back a grade. Imagine the outcry. There’d be blood in the streets, and I am not being hyperbolic. People would simply not accept the idea that half the kids in the city are unprepared for the next grade.
Now, let’s think about teachers. This is trickier. It’s not as easy to find information on the exact passing percentage of different teachers’ classes. Instead, let’s think for a moment about the factors that might influence any given teacher’s proficiency rate. Imagine, for a moment, that you teach a self-contained special education class. Your proficiency rate is going to be low. It is going to be low because your students have disabilities, and those disabilities are probably going to be reflected in their scores no matter how great you are at your job. That used to be a common-sense proposition.
Now, let’s imagine that you are a teacher with a high percentage of English Language Learners in your class. Your proficiency rate is probably going to be low because your students are trying to learn a new language on top of learning the actual content of your subject, and that’s not easy. Also, the ELA test has to be taken in, you know, English. Those kids will probably do poorly even if you are an amazing teacher. That used to be a common-sense proposition as well.
Now, let’s imagine that you teach 7th grade English in a school where the majority of students failed the 6th grade ELA test. They also failed the 5th grade ELA test. They failed 4th grade, and they failed 3rd grade. They weren’t tested before that, but you know from their 2nd, 1st, and kindergarten teachers that they weren’t performing up to grade level back then either. Your proficiency rate is probably going to be low, because a majority of your students were behind from the moment they walked through your door. If a student walks into your 7th grade class reading at a 4th grade level (as many do) and walks out reading at a 6th grade level, then you have done your job. That kid has learned, and you have taught. He’s still going to fail that 7th grade test.
Now, let’s imagine that you teach 7th grade English in a school where the majority of students passed the 6th grade test. They also passed the 5th, 4th, and 3rd grade tests, and their kindergarten teachers tell you that they walked in the door on their first day already reading ‘see spot run’. Unless you seriously screw up, they will probably pass that 7th grade test.
So, you can’t fire teachers based on their students’ test scores because you’d end up firing a whole slew of awfully good teachers who accomplished a lot, given the raw material. You can’t promote or reward teachers based on their students’ test scores because you’d end up rewarding a whole lot of teachers who are quite good, but not any better than those peers who had less raw material.
Making employment or promotional decisions based on the tests would require the educational system to either engage in some outrageous injustice, or admit some very uncomfortable facts.
If more than half of the city’s students fail these tests, if they fail despite all the hours and all the money and all the effort that their teachers and their schools pour into teaching them—then maybe some of these students are just not going to pass. Maybe some of these students are so hampered by their early childhoods or their home environment or their IQs that even heroic efforts won’t be enough. Using test scores for promotional decisions would drag these speculations into the light, and we can’t have that, so we don’t use them for promotional decisions.
If there are many dedicated and hard-working teachers who design careful lessons and still have a high failure rate, then maybe teachers aren’t the sole or the primary determinant of student success. Maybe other factors outside the classroom have an equal or greater impact on students’ ability and willingness to learn. Firing teachers based on test scores would bring these questions to the fore, and we can’t have that, so we don’t use them for employment decisions.
We can’t admit that all this sturm und drang about test scores was useless, though, so we’ve got to use them for something. We’ll use them to make decisions about schools. Average school scores are far enough removed from individual schools and teachers that we can talk about them without asking uncomfortable questions.
This is an anonymous blog, but I’m still tap-dancing around the issue because it is that radioactive.
What if some students just can’t learn some things? What if there are some students who aren’t likely to ever be ‘proficient’, even with good teachers and well-planned lessons? Let’s not even touch on the question of why (in this post, anyway). Let’s not even ask if students are disadvantaged by their home environment or their social-emotional skills or their word deficit or their IQ. Even the barest suspicion that some students might be unlikely to learn, for whatever reason, would require us to revamp all of our assumptions about how education works and about who is to blame when it fails. It would require us to confront the idea that education, like life, is not and cannot be equitable.
So what do we do, if we are DOE officials? We screw around with data and our use of it, so that we don’t have to ask uncomfortable questions. It’s easier to close down a school and make appropriate mouth-noises about accountability and equity than it is to fail thousands of students and admit that, for unknown reasons, they don’t seem to learn much.
Aye Carranza! 