If you are anxious to judge a teacher on test scores...

[jpgray]

Please read the following (VAM refers to value-added modeling, the sort of model used to rate teachers by student test scores):

…VAM estimates of teacher effectiveness should not be used to make operational decisions because such estimates are far too unstable to be considered fair or reliable.

-National Academy of Sciences

VAM results should not serve as the sole or principal basis for making consequential decisions about teachers. There are many pitfalls to making causal attributions of teacher effectiveness on the basis of the kinds of data available from typical school districts. We still lack sufficient understanding of how seriously the different technical problems threaten the validity of such interpretations.

-Policy Information Center, Educational Testing Service

The estimates from VAM modeling of achievement will often be too imprecise to support some of the desired inferences…

The research base is currently insufficient to support the use of VAM for high-stakes decisions about individual teachers or schools.

-RAND Corporation

The above were referenced in the Economic Policy Council's report, Problems with the Use of Student Test Scores to Evaluate Teachers. In the study, the authors argue:

For a variety of reasons, analyses of VAM results have led researchers to doubt whether the methodology can accurately identify more and less effective teachers. VAM estimates have proven to be unstable across statistical models, years, and classes that teachers teach. One study found that across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40%. Another found that teachers’ effectiveness ratings in one year could only predict from 4% to 16% of the variation in such ratings in the following year. Thus, a teacher who appears to be very ineffective in one year might have a dramatically different result the following year. The same dramatic fluctuations were found for teachers ranked at the bottom in the first year of analysis. This runs counter to most people’s notions that the true quality of a teacher is likely to change very little over time and raises questions about whether what is measured is largely a “teacher effect” or the effect of a wide variety of other factors.

That the LA Times ignored such caveats and made freely available teacher rankings based on VAM, possibly driving one teacher to suicide, may be the most disgusting display of technocratic witch-hunting I have ever witnessed. Do not be a party to it without researching the statistics some are so anxious to wield as a political weapon.

[MineralMan]

1. Your last paragraph is woefully incorrect.

That teacher was far from the worst in ranking according to test results. Further, the LA Times posted a complete listing of the results for all teachers in the district, but only on their website. The teacher who took his own life was never "singled out" by the LA Times, nor was he listed as "the worst in the city." Indeed, he wasn't even the worst ranked in the school in which he taught.

Unrecced for misrepresenting the facts.

[jpgray]

2. Edited to fix. One of my bookmarks must have had it wrong

Thanks for pointing out the error, and sorry if anyone was misled.

[MineralMan]

3. Thank you for making that correction.

[dkf]

4. Apparently any testing or assessment is pointless, including students test scores.

I wonder how many parents would accept this if their kid came home with an "F" and they tried to explain how testing methodologies are crap.

The hypocrisy is breathtaking.

[jpgray]

5. This is more akin to judging the parents based on the kid's F

Can you tell the quality of a parent from a child's grade? It's silly to argue it says nothing about the parent, but it is equally silly to claim it says everything, and it may be wrong to claim it says something significant. Say a child goes from a C average to failing--is it the fault of the parent? Are those parents whose kids drop in GPA most over the course of say, two years, necessarily the worst parents?

[dkf]

9. Teachers say the kids are failing because of bad parenting.

Funny how that goes only one way.

And yes...unless your child has a learning disability their grades are a reflection of their parents.

[zipplewrath]

8. What is most telling

You won't find alot of top ranked private schools, especially on an international scale, that use this kind of teacher evaluation. And that is even true of Asian cultures which place alot of burden on the teach for the student to succeed. Testing, or student performance, might be used as a basis for reviewing a teachers performance, but it would be the result of that review, and not the test scores themselves that would be judged.

[dkf]

10. Internationally they probably don't need to justify firing a teacher based on any scores.

I imagine they have zero tolerance for less than excellent teachers.

[zipplewrath]

11. But it's how it's determined that is the point

They don't use arbitary testing results to identify the "less than excellent" teachers.

[dkf]

13. They probably just use student test scores.

What else can you use that works across the board?

[zipplewrath]

17. No, they don't

Go to any prep school, or international school. If there are under performing students, they check lesson plans, they check room discipline, homework assignment leves, and same student performance in other classes. I've gone to schools that had very extensive teacher evaluations. There were known "hardasses" that often generated lower test scores than others. They stayed around. In fact, it was the "cupcakes" that could often find themselves in trouble. "Teach to a tes" was a good way to get in a whole heap-o-trouble.

[dkf]

22. In Asia?

That is all I'm concerned about.

[Hannah Bell]

19. lol. tain't so, mcgee. i saw plenty of "bad teachers" in japan.

i call teachers who stand in front of a class of 50 high schoolers, droning on & on as students sleep behind their books "bad".

[exboyfil]

12. I think private schools do use

college placement tests in evaluation (the same way that it is emphasized in England).

I watched my older daughter's ITBS drop significantly in two areas in which she had substandard teaching (she got high As in both classes). In one subject the teaching effort was pathetic (he really wants to be coaching and not teaching). Unfortunately, my daughter had this coach for two years. She now has a very good English teacher (her best teacher this year). The other subject, Social Studies, was not taught badly - just mediocre when compared to her Science and Math teachers.

I am Homeschooling my youngest in English and Social Studies this year in response to the English teacher's failings. I could not take just one subject so I bundled the two and integrated the curriculum.

I will be curious to see if my younger daughter drops in performance after a year in Homeschooling when compared to my older daughter.

My perception of the curriculum and teaching performance came before the test results. I know what to expect from these classes, and the English teacher's performance was far removed from my expectations. His only feedback on grading for my daughter was to look at the grades.

Standardized tests get a lot of criticism, but at least in my daughters' case I do see correlation in performance. My older daughter actually became a worse writer after two years with the above mentioned English teacher. She has blossomed with her current English teacher. Unfortunately she has gone from two top notch Science teachers and a superb Math teacher last year to mediocre ones this year. It is unfortunate that our standardized tests are in November so I really can't say if my older daughter's scores will change this year. I know her writing has gotten much better since the start of the year.

At first glance I though the value added approach for assessing teachers using standardized testing appeared to be a good approach. I am not so sure now given the criticisms leveled. I would expect to look at trends in performance - one year is meaningless, but if a teacher consistently has children underperform on tests when compared to other teachers, then it should be part of the assessment. I have formed opinions on teachers' performance and later seen those opinions correlated to the test scores.

What accountability measures should be put in place for teachers? The same kids that can write cogent essays also seem to be the same kids who do well on the standardized tests in English. As far as Math goes standardized tests are directly on point. Do tests like ITBS really do that bad a job of assessing childrens' capabilities? At least in my daughters' case, they have done a very good job in assessing strengths and weaknesses.

[zipplewrath]

18. Depends upon the grade level

At lower levels, there's not alot of standardize testing in use. The predominate reality is that the next years teacher knows what's being accomplished by the previous years instructors. And that goes further into issues of the writing skills being seen in the social studies classes and such. The teachers know who is doing what, and who is not doing what.

It's been tried in a few areas, but administrations don't like it. But you can basically have "teachers counsels" which will rate teachers and hand out performance pay bonuses. They work, even if not perfect. Yes, there's always potential politics, but there's even ways to have appeals and reviews to avoid much of it. But the teachers know who the good ones are, and who the slackards are. Empower them, and they'll take care of it.

[Smarmie Doofus]

6. Rec. So a reasonable person might be led to ask,.....

.... "Why are they being used incorrectly and unethically?"

Hey.... maybe there's a *political agenda*.

"Is it possible, even *conceivable*...." ( tee hee... a line from a movie.)

[hfojvt]

7. I only heard of "value added" last month

and immediately saw a flaw.

As it was described to me via the Daily Howler it rates teachers based on student percentiles. That is, if a student was testing at the 60th percentile and at the end of the year is testing at the 70th percentile the teacher gets credit for the "value added."

The problem I see is that for every gain there must be an equal loss. That kid who goes from 60th percentile to 70th percentile has just bumped some other students down. The kids who used to be at 61st, 62nd, etc. just got bumped down by the kid who passed them.

The analogy I used to illustrate it was track. Suppose kids are ranked by their time running the mile (see how old I am, I say mile instead of 1500 meters), and look at three coaches and three kids. Runners are timed at the start of season and at the end of track season and coaches are evaluated by "value added".

******* 1st time **** end time
Coach A *** 7.28 *** 8.02 - - a bad coach since his runner went from 1st to 3rd
Coach B *** 7.44 *** 7.59 - - a neutral coach since his runner stayed at 2nd
Coach C *** 7.53 *** 7.58 - - a good coach since his runner went from 3rd to 1st

and yet all the kids got slower and still one coach is labelled as good

Coach A *** 7.51 *** 6.12 - - bad, went from 1st to 2nd
Coach B *** 7.57 *** 6.13 - - bad, went from 2nd to 3rd
Coach C *** 8.04 *** 6.11 - - good, went from 3rd to 1st

two bad coaches and only one good, and yet all the runners showed dramatic gains in their times

The other trick is that, it is better for the coach to have the slowest runner. Coach C has nowhere to go but up. The other thing is, if there is only one teacher teaching all the students, then your total net "value added" is always going to be zero. If the kids are initally ranked abcde, and then are ranked acbde the gain by student c is negated by the loss of student b. That is true for every permutation. Go to bcdea and the four +1 steps are negated by the -4 of student a. Goto baecd and the gain of b is cancelled by the loss of a, the gain of +2 by e is cancelled by the losses of -1 by c and d.

It's a textbook zero sum ranking system, and learning is not zero sum. Students usually make huge strides between kindergarten and their senior year.

[exboyfil]

14. I think these tests are normed to some prior

fixed performance point (an earlier group of students). In that way they are a standard. Eventually the norm has to be shifted, but it appears to stay around for some time.

I know that CogATs are done this way. Not sure about other tests.

For simplicity it is reported as a performance percentage associated with the current peer group, but you can see that this cannot be the case since that peer group has yet to be completely formed (not everyone takes the tests at the same time).

[Recursion]

15. If the question were percentile, yes

But for once we've broken free of the demonstrably false idea that every test score follows a normal distribution. The VAM systems I've seen don't care about the child's placement in his cohort but about his mastery of the material being tested. eg, if the top-testing kid only gets 25% of the questions right, that's still a failure.

[madfloridian]

16. Because I was a good teacher, I was given many challenging students.

I had classes some years who could not score well on tests no matter who taught them.

At one school in a comfortable middle class neighborhood with involved parents, most classes had good test scores and made the teachers look good.

At the last school where I taught the scars of their home life got in the way too often. They were just as capable as the kids in the other school overall....but they might be in our school 3 or 4 days or a week or so and then gone.

Their parents were often not involved very much, and it showed.

No matter how many years you cover of scores of a teacher, it is still not fair. It does not take into consideration all of the factors involved in learning.

The head of the L. A. schools was afraid to say many good things out loud about Mr. Ruelas who took his own life. He said the teacher evaluations were private. He did speak out on it though, and he admitted that he had a "great performance review" by supervisors.

Well, he could have made an exception in this case and said it much more publicly than he did. As Diane Ravitch said:

Last week, I was in Los Angeles. I spoke to L.A. teachers, who were shamed by the Los Angeles Times' disgraceful release of test-score data and ratings of 6,000 elementary teachers as more or less effective. I had previously believed that such ratings (value-added assessment) might be used cautiously by supervisors as one of multiple measures to evaluate teacher performance.

The L.A. Times persuaded me that the numerical scores—with all their caveats and flaws—would drown out every other measure. And, in fact, the L.A. Times database contained only one measure, based on test scores. And so I concluded that value-added assessment should not be used at all. Never. It has a wide margin of error. It is unstable. A teacher who is highly effective one year may get a different rating the next year depending on which students are assigned to his or her class. Ratings may differ if the tests differ. To the extent it is used, it will narrow the curriculum and promote teaching to tests. Teachers will be mislabeled and stigmatized. Many factors that influence student scores will not be counted at all.

The Democratic party scorns public school teachers and public education at a big risk to themselves and to the country.

[YvonneCa]

23. EXCELLENT post, especially your last...

...sentence. The risk Democrats are taking saddens me greatly. The even GREATER risk of not getting education right for the country and for the next generation saddens me most of all.

[TexasObserver]

20. I am only anxious to end this tyranny of the testing.

It's an invalid method of judging school teachers.

[DailyGrind51]

21. In Illinois, teachers must pass three state exams prior to certification.

Basic Skills, Content Area (Elementary Ed., Math, Language Arts, Reading, Social Studies, etc.), and an Aptitude for Professional Teaching Test.