Attention DU lawyers and/or educators: Test scores and potential criminality.

[Smarmie Doofus]

Are there any statutes on the books that protect the general public against this sort of thing? It is so *blatantly* fraudulent as it denies the general public accurate info on which to base its educational and political decisions.

To get to the gist, look at the 8th grade math scores ( can't seem to transport the image but it's on the linked site in the form of a line graph.) NYC kids scored a four point proficiency improvement on the NAEP ( gold standard; this is not disputed anywhere that I know of) and on the NYS/NYC tests the same 8th graders jumped *26* ( twenty six) points.

In the same two year period of time. (2007 to 2009). During the same two year period Mayor Bloomberg ( education reformer, data-wonk, school choice advocate, blah, blah, blah) ran a (abortive) campaign for president of the US and a ( successful) campaign for reelection as Mayor. ( The NAEP scores for NYC kids were released just after the election. How... everybody together, now!.... conveeeeennnieent.)

Here's my question: this is *legal*? And if there *is* some question about it's legality, under whose jurisdiction would it fall? In other words... who should be investigating? ( And if the answer is the US DOE just send me some cyanide pills or enough rope with which to hang myself.)



>>>>>http://www.uft.org/news/teacher/top/math_tests_no_improvement_since_2007/>>>>>

Federal math tests find no improvement in city since 2007

by Maisie McAdoo

Dec 17, 2009 12:25 PM

Despite a relentless focus on test performance since Mayor Michael Bloomberg took over the school system, New York City failed to score any significant gains in either 4th- or 8th-grade math in the latest round of the National Assessment of Educational Progress, the gold standard national achievement test.

The findings directly contradict results on the New York State tests, which show city students making large performance gains since 2007.

The city’s 4th- and 8th-graders did make progress from 2003 to 2007, the national tests show, but the momentum flatlined after 2007. In addition, on the closely watched racial achievement gaps, the city has not narrowed the black-white or Hispanic-white gaps in math since 2003, the NAEP results revealed, though the mayor and Chancellor Joel Klein have made heavy investments in trying to close the gap.

“We have heard endlessly from the administration that the sole reason our students gained ground was because of the city Department of Education’s test-focused approach to education. They led the public to believe that in fact we were closing the achievement gap. Clearly that is not the case,” UFT President Michael Mulgrew wrote in a letter to members on Dec. 8, the day the results were announced.
A more reliable test

The NAEP, designed and administered by the federal Department of Education, is widely considered the gold standard measure of student achievement, both because it is so rigorous and because it is far less subject to political pressures than state achievement tests. NAEP tests challenging skills and knowledge levels that a national panel of experts believe are appropriate at each grade.

On the NAEP exams, New York City 4th-graders gained one point in proficiency since 2007 while the city’s 8th-graders rose four points.
In contrast to the NAEP results, the city’s recent state test results showed 4th-graders gaining more than 10 points and 8th-graders a whopping 25 points in proficiency since 2007.

Such unusual leaps have led State Education Commissioner David Steiner and Regents Chancellor Merryl Tisch to call for an overhaul of the state tests and resetting the proficiency bar.

They have also led the UFT once again to challenge the city DOE’s reliance on these tests.

“The administration has made this flawed state test data the key barometer for everything from school progress reports and teacher bonuses to school closings,” Mulgrew told members in his letter. “Now it even wants to use this data to decide teacher tenure. But its approach is fundamentally flawed, because state test results are not aligned with true learning.”
Spinning the results

Chancellor Klein, in a presentation to reporters, argued that the city’s small upticks showed the city making progress from 2007 to 2009. But NAEP determined the narrow gains were “statistically insignificant.” Still, the chancellor went several rounds with skeptical reporters in trying to make his case.

The chancellor’s office also privately commissioned the National Center for Education Statistics to pull apart the city scores from a “rest-of-state” average for New York. (A rest-of-state average removes the one-third of students from the state total who are city students.) An elaborate PowerPoint, which his office prepared for reporters, showed the city outperforming the rest-of-state average on NAEP over the last four years.

Other DOE PowerPoint slides showed the city’s black and Hispanic youngsters narrowing performance gaps with their suburban counterparts in some instances. “Whose blacks are on top?” the chancellor demanded, showing reporters the slides.

But the reporters did not seem persuaded. The chancellor did not present the underlying data for his rest-of-state comparisons, and his claims that city minority students were closing in on suburban minorities could not be fully verified.
An urban district focus

The city’s performance on the NAEP is part of a federal focus on urban school districts that began in 2003. Before then, NAEP results were only reported for entire states.

............snip........

But the city has scored higher than the urban district average since TUDA began, testament to its well-educat

more at link ( including line graph of 8th grade math tests)

[AngryAmish]

1. what?

[Smarmie Doofus]

2. Fraud. Or the appearance thereof. nt

[AngryAmish]

3. in the test scores?

[Smarmie Doofus]

5. Indeed. nt

[FBaggins]

4. Convenient, yes. But what would be the alleged criminality?

Is your concern the difference in state/federal measurements? Or is it the timing of the release to occur after voters would have the ability to act on the information?

What influence would the NY mayor have over the release of federal data?

[Smarmie Doofus]

6. That's what I'm asking.

>>>>Is your concern the difference in state/federal measurements?>>>

Yes


>>>Or is it the timing of the release to occur after voters would have the ability to act on the information?>>>

Not so much. Though that's interesting also.

>>>>What influence would the NY mayor have over the release of federal data?>>>

Not sure. He's got a lot of influence in the Obama administration and in the Duncan dept. of ed.

[FBaggins]

7. So are you saying that the numbers may have been manipulated...

...or is it the validity of the assessments themselves?

For my part, I can easily two assessments diverging by that much without being invalid... but would prefer to review the actual data.

Absent evidence that both assessments were valid but the NY data was somehow manipulated (or that the NY test was intentionally mis-contructed)... I don't see much "there" there.

[Smarmie Doofus]

8. How? They are measuring the same kids on the same material.

>>>>>For my part, I can easily two assessments diverging by that much without being invalid... but would prefer to review the actual data.>>>>>>

4% improvement vs. 26% improvement. This doesn't raise a red flag?

>>>>So are you saying that the numbers may have been manipulated...>>>>

A distinct possibility.


>>>>.or is it the validity of the assessments themselves?>>>.

Again. Quite possible. I work in the system. Many educators have shared with me their opinion that the city/state test has been progressively "dumbed-down". Does that go to "validity" ?

[FBaggins]

11. That's an assumption.

They aren't necessarily assessing the same kids on the same content. And they don't necessarily (in fact don't IIRC) use the same standards for proficiency.

The article also makes the mistake of calling one the "gold standard" because it is less open to political manipulation. I wouldn't say that's the case at all. BOTH tests are subject to political problems. The federal government has reason to display lower scores to increase their own influence in local affairs, while the states have reason to display higher scores to claim that they're doing a good job.

The key here would be to review the testing methodology.

[Smarmie Doofus]

14. And this would require investigation, it seems to me.

>>>>The key here would be to review the testing methodology.>>>>

>>>>The federal government has reason to display lower scores to increase their own influence in local affairs,>>>

To each his own... agenda-wise. Note however... that the NAEP scores for NYC 8 grade kids are *stable* over time ( 21 to 26 over a six year period.). Suggesting a more modest political goal than one that might be entertained by a wildly ambitous NYC CEO who finds himself in charge of a school system

( 34.4. to 71.3 over the same period.)

Come on, now.

[FBaggins]

13. Update - As I suspected... the results are different because they don't test the same things

One example I found was that the 4th grade NAEP contains algebra and geometry as 15% of the test. Students in NY are regularly introduced to those topics in 5th grade... so you could expect them to miss a high percentage of those questions. If "proficiency" were set at, say, 75%... the NY 4th graders, would have to top 87% on the remaining portions of the test in order to be proficient.

Now... it's perfectly reasonable to debate whether these concepts appropriately belong in 4th grade or 5th grade... but testing based on those opposing assumptions can give you apparently out-of-whack results that have more to do with the long-term curriculum decisions than with the educational success of the students in question.

It's like I implied in the prior post. This looks like it represents a tug-of-war between federal and state control over education. If you judge kids by different standards, don't be surprised to get different results. NY could dramatically improve their NAEP scores (in this 4th grade math example) by "teaching to the test" and spending a couple weeks each on algebra and geometry... but it would be an artificial bump.

[Smarmie Doofus]

15. I'm not comparing the NAEP scores with the NYC/NYS scores;

I'm comparing the dramatic, election year *improvement* of the city scores with the tiny, statistically "insignificant" improvement on NAEP over the same period of time.

Whether the NAEP acores are higher generally or lower generally is beside the point. The suspicion of illegality ( if there are in fact statutes that pertain to this sort of thing) is related to the discrepancy in the *rate of improvement*.





>>>>NY could dramatically improve their NAEP scores (in this 4th grade math example) by "teaching to the test" and spending a couple weeks each on algebra and geometry... but it would be an artificial bump.>>>>

Without a doubt. And they could artificially bump the city/state scores in exactly the same way.

[proud2BlibKansan]

9. NAEP scores are not valid

They aren't used for anything other than headline stories. Also NAEP only tests selected kids. I am a special ed teacher and when our school had to give the NAEP a few years ago, we were told there could be NO accommodations. So we didn't test any kids who had IEPs. We didn't test any of our ELL kids either.

So just ignore the NAEP.

[FBaggins]

10. I thought you said that never happens???

You don't see that as an incentive to mis-label an under-performing student as special needs?

A school can artificially adjust their test scores by simply labeling their under-performing students as disabled and then not including them in the test. Schools are judged (not that they necessarily should be, but we know they are) in many cases by those scores. You don't see the incentive?

[proud2BlibKansan]

16. No since that is highly illegal and the state monitors our testing it isn't going to happen

Sure there's an incentive but you risk losing your professional certification. The state sends monitors into our schools and triple checks all of our testing data so it would be hard to get away with anyway.

[FBaggins]

18. Nonsense. It happens all the time.

Identification of learning disabilities is not a hard and fast designation. There's judgement involved. So there isn't any way for the state (which, BTW, also benefits from that incentive)to accuse you of falsely labeling a student as special needs. If by "monitoring your testing data" you mean that you couldn't impact your test scores in this way... you just admitted that you do it. So I don't see the point.

I know several teachers here in NC who say that it happens regularly. I also know of a number of students who were labeled while in school but have outperformed in one-on-one instruction. in those cases of course, it's possible that the child was disabled but had a disability that didn't have an impact in an alternative environment.

[proud2BlibKansan]

19. No it doesn't - this is just another myth

I could argue the opposite, that many kids who are indeed LD are not identified because the criteria changes all the time. It gets much harder to qualify them from year to year.

And like I said, if you break the rules, the state is breathing down our necks just waiting to penalize us. Anyone who claims the opposite isn't working in the same field I am. :)

[FBaggins]

20. That's not "the opposite".

I have no doubt that there are disabled kids who never get properly identified as such because they are presumed to be a discipline problem... or simply because the staff at the school is less well trained at identifying disabilities. That doesn't mean that some schools don't err in the other direction (and live in a system that incents them to do so). The number of kids who are labeled "disabled" is up dramatically over the last few decades. There is plenty of room for both mislabeling (intentional or otherwise) AND picking up kids who previously fell through the cracks.

You claim the state (which, again, has an incentive to allow this to boost test scores) is "breathing down your necks"... but you can't demonstrate HOW they would ever catch you.

Do you deny that school administrators all across the country play games to artificially boost test scores? You just don't accept that THIS game is not beneath them?

Do you deny that I previously demonstrated a case where it most certainly DID happen and other counties in the state lost millions in spec-ed funding because of it?

[proud2BlibKansan]

21. The state checks all of our paperwork

They have employees whose only job is to check up on how we qualify kids. No way could we EVER sneak one past them. So they would indeed catch us if we were qualifying even one kid who wasn't really disabled. They kick back our paperwork all the time. It's actually quite annoying.

I can't speak to how your state monitors its disability qualifications, since I don't live there. All I can say is no way can we get away with it here.

Sure admins play games to boost test scores. But only the games they can get away with. And since the state sends people to monitor us while we test, this is harder also. And again, I don't know if your state does this or not.

Here's an example: we have to take down all charts and graphs, etc that the kids could use while testing. Multiplication tables, etc. And those are present in nearly every classroom. But during the state tests, they have to all come down. And several schools get in trouble every year for not stripping charts off their walls. I know other states don't have this requirement because I have been to national conferences where teachers are taught how to help kids make cheat sheets to use while testing, like multiplication charts.

It's all pretty stupid because kids in 6th grade and above are allowed to use calculators on state tests. So they have to take down the charts but they can just use the calculator for the multiplication facts they don't know.

[FBaggins]

22. And the "paperwork" PROVES whether a child is disabled?

There's some blood test you give that never gives a false positive and the state can check that?

You can continue to claim that it can't happen.. but you have yet to give a reason that boils down to more than "because I said so".


I honestly don't mean to be offensive, but it's simply the case that without ANY cheating at all, you can have two kids next to each other who score low on a particular test and have one of them be disabled and the other one isn't... and you can easily have two professionals disagree as to whether one or both is mildly disabled. There is no way for a state to challenge this determination (nor would they be qualified to do so in most cases).

As I've said before, it doesn't have to be at all sinister. It can be done with the very best of motives. Take a look at the minority disability rate for your state. Does it reflect a "truth" that black kids are more likely to be disabled than white ones? Or does it show that they are just more likely to be labeled as such?

If a school has a spec-ed program for kids with reading difficulties, does not that program aide the non-disabled non-reader almost as much as the disabled one? If Johnny is having difficulty reading and there are no programs that he would otherwise qualify for... what harm would a teacher necessarily see in labeling him in a way that gets him the help he needs? And, more importantly to this discussion, what state official could possibly gainsay the determination?

As for the your last couple paragraphs. I agree. though I don't see how you can doubt that a principal who would pull such games to manufacture higher test scores would DARE labeling her ten lowest-performing non-disabled students as disabled so that their test scores don't count against her.

[proud2BlibKansan]

23. I guess you don't understand how kids are labeled as disabled.

They are given IQ and achievement tests. All standardized and individually administered by a registered psychologist or psychological examiner. For a child to be labeled as MR (retarded) in my state, their IQ needs to be below 70 with adaptive behavior scores also that low. (Adaptive behavior is how well one functions in the world, so for kids it's a test of self help skills, from shoe tying to toileting to handling emotions.)

For a kid to be labeled as LD, there needs to be a difference of 22 points between IQ and achievement. So if your IQ is 100 and you score a 78 in Reading, you have a learning disability in Reading.

THAT IS ONLY IN MY STATE. I can't speak specifically to other state guidelines.

Now the dumb part is the discrepancy for LD used to be 15 points. So we have some kids who would have qualified a few years ago but don't qualify now since the discrepancy is bigger. Also some of our kids will qualify in other states because those standards are different.

So you can be labeled as disabled in one state but not in another. And that's downright dumb.

But to reply to your statement, yes states can and do challenge our 'determinations'.

=================

Yes special ed programs benefit non-disabled kids too. There are pushin programs where sped teachers go into gen ed classes and work with both sped kids
and other kids. It is also beneficial to remove sped kids sometimes for pullout classes.

[tonysam]

24. It is similar in Nevada. n/t

[proud2BlibKansan]

26. Now. Once they ARE labeled as disabled, the game really changes.

I am reminded of the discussion we had about admins wanting us to give alternative tests to even more sped kids.

[tonysam]

28. And of course it is illegal under NCLB to put LD kids on alternate testing

Not that it mattered to the principal who tried to con me otherwise.

[proud2BlibKansan]

30. They really play around with the MR kids too

They change the criteria every year. One year they had to be nonreaders. Next year they could read but they had to be 3 years or more below grade level. High functioning MR kids are being bounced on and off alternative testing. It's sad.

[tonysam]

31. That's something how they bounced around kids diagnosed as MR

They shouldn't be taking standardized tests, period, if they are truly MR. I did have kids who were clearly misdiagnosed as MR or were allowed to take alternate testing when in fact it was obvious they weren't eligible. I and the IEP team changed those kids over.

[proud2BlibKansan]

32. I think kids should be allowed to be excluded

Parents should be able to request their kids NOT be tested. We don't need to test EVERY kid. We can do a sampling group and still get valid data.

A couple years ago, a school I know of had a student who was terminally ill and hospitalized during testing. The school was penalized and didn't make AYP because he wasn't tested - but there was no procedure for excusing a kid from being tested.

Still blows my mind. The idiots who wrote this law never saw the possibility of a kid DYING and not being able to test. :mad:

[tonysam]

38. My understanding is kids could be exempted if parents request it

I know of one kid who didn't qualify for alternate testing because his eligibility was changed, yet his physical disabilities were such (he had CP) that using a computer, a scoring sheet, or even giving oral answers would be difficult.

[proud2BlibKansan]

39. Not any more

We cannot exclude ANYONE. Like I said, even a kid dying in a hospital was expected to take the damn test.

But yes, a few years ago we COULD exclude a few kids.

[FBaggins]

27. I understand just fine.

I also understand that a disability isn't the only reason that someone could have a gap between measured IQ and achievement. It's recognized that there are flaws in that system of identification and some ares also use "Response to Intervention" instead (or as well). This so-called "discrepancy model" has plenty of problems. It's also the case that this (as I implied) isn't a hard/fast standard. Two professionals assessing the same kids can STILL come to different conclusions about them.

These also aren't the only special-ed designations, are they? How do you identify ED? Autism? ADD?

Imagine a non-disabled child who simply isn't performing. Rather than demonstrating the rigor of your process, you've just demonstrated how EASY it is to mis-label that child. His achievement is obviously below what his IQ implies it should be. Even if he isn't disabled, your one identified gatekeeper is satisfied.

The CEC itself is dissatisfied with these two options but admits no others have advanced enough to be superior. If THEY can't agree that child can definitively be identified properly... the state most certainly can't do it.

So you can be labeled as disabled in one state but not in another. And that's downright dumb.

That's because the system itself is disabled. It's so hamstrung with bureaucracy that learning suffers.

[proud2BlibKansan]

29. Oh I agree the system is messed up.

My point is that it's not as easy to game the system (at least not where I work) as one might think.

The other disabilities are labeled just as rigorously as the ones I described. An ED label requires medical documentation. I think autism does too. ADD is not a recognized disability. ADD kids can have modifications with a 504 plan. A 504 is a legal plan that outlines adaptations for kids with short term or other disabling conditions.

[FBaggins]

33. Didn't you just document how easy it is?

Let's go back to an imaginary student. Perfectly normal kid (100 IQ to boot)... but he drew the unlucky card. He got the worst teacher in the worst school at every grade level from K-4. To make things worse... his parents either don't care, or aren't capable of helping.

Neither bad parents nor bad teachers are "disabilities" - but can we really doubt that he'll be at least 22 points down on the achievement test vs. his IQ?

When this kid is labeled disabled so that he can get the help he needs (even if he doesn't technically qualify for it)... how is the state going to stop you?

[proud2BlibKansan]

34. You're asking for a purity test

Until we can examine a brain and have a definitive characteristic to look for - then labeling kids as disabled is a crap shot.

But with the current system - the ONLY one we have here - a kid with a 22 point discrepancy is indeed considered LD.

[FBaggins]

35. :) That's what " No way could we EVER sneak one past them" requires?

Now we've demonstrated that this isn't the case.

labeling kids as disabled is a crap shot.

Isn't that what I've been saying? So there's no way for the state to keep you from designating an under performing child as disabled.

But with the current system - the ONLY one we have here - a kid with a 22 point discrepancy is indeed considered LD.

And yet in reality, they aren't. This has been my point for weeks now. There ARE under performing kids who are NOT disabled yet who CAN be labeled as disabled. We've demonstrated that it CAN happen... I've documented cases where it HAS happened... and I'm sure we both agree that there are some school administrators who wouldn't let ethical considerations stop them. What's left to debate?

I'm off to spend the rest of the year (love the sound of that... wish it weren't late December) in Colonial Williamsburg with the kids. I'll check in later. Happy New Year!

[proud2BlibKansan]

37. As I have said repeatedly,

I can only discuss in detail the process in my state, and no, we are not able to label kids as disabled unless they really are disabled AS DEFINED BY THE CRITERIA SET BY THE STATE.

I cannot discuss the process in other states where it may be possible for non disabled kids to sneak through the cracks. But in my state, no, it's as close to impossible as it can get.

[proud2BlibKansan]

25. I should have added:

In KS kids are labeled as LD based on standardized tests and 'teacher judgement'. At least that's what they were doing a couple years ago.

It's hard enough to keep up with state regs in one state so I don't pay a lot of attention to the others. But I do know that many are less stringent than my state so it may be easier to label kids as disabled in those states.

[Smarmie Doofus]

12. That would appear to be a minority view:

From wiki:>>>>Since NAEP assessments are administered uniformly to all participating students using the same test booklets and identical procedures across the nation, NAEP results serve as a common metric for states and selected urban districts that participate in the assessment. The assessment stays essentially the same from year to year, with only carefully documented changes. NAEP reports results at the national level and provides state results for some assessments. On a trial basis, NAEP has released the results for a number of large urban districts>>>>>

Also... NAEP's validity is not disputed by Bloomberg's Chancellor Klein and the embarrassing gap between NAEP and NYS/NYC scores prompted the new NYS ed commisioner David Steiner to call for an "overhaul" of the *state* tests... which are his "own" tests ( point of info:he's new to the job)... not an overhaul of *NAEP*.

*Overhaul* is fine. I'm just wondering if that shouldn't be preceded by some fact-finding ( investigating, if you prefer) to discern how the city/state testing "miracle" of 2007-09 actually occurred.

Finally .... the exclusion of ELL and iep kids from NAEP , as you describe, would tend to artificially *raise* the NAEP scores. The problem for Bloomberg et al is to account for why the NAEP scores are so LOW.

[proud2BlibKansan]

17. But it's true. It's not rocket science to understand a test that excludes kids is not valid

I don't know about NY but in my state we give the NAEP after a month of the mandated NCLB state testing. The kids are tired, the teachers know the NAEP means nothing and no one really cares about the scores.

[hughee99]

36. Is this really any different then announcing

unemployment or economic numbers that are wildly optimistic as factual and then eventually "revising" those numbers significantly downward at some later, more convenient time?