Big jumps in test scores among groups of students at 93 Arizona schools should have prompted education officials to question if educators were violating testing rules, according to a newspaper analysis reviewed by testing experts.
The Arizona Republic and USA Today found that average scores in at least one grade in each school took a one-year jump that testing experts say is statistically improbable. Experts told the newspapers that at least one group of students at 22 of the schools failed to achieve similar scores the next year.
That’s a sign that the prior year’s tests merited more investigation, according to experts interviewed by the papers. The odds of those jumps occurring by chance ranged anywhere from 1 in 370 to 1 in 396 million.
Arizona Attorney General Tom Horne and current state superintendent John Huppenthal said they don’t believe that test-giving improprieties are widespread. Horne was schools chief during the five years reviewed by The Republic.
The anomalies, found using a methodology widely recognized by mathematicians and testing experts, are not proof that any cheating on AIMS by educators occurred at those schools. The Republic contacted officials at more than a dozen schools with especially large gains, and all insisted the increases were real.
Stepped-up scrutiny of tests is more imperative than ever because the incentive for educators to manipulate scores is only growing, experts say. Low scores on achievement tests increasingly can affect teachers’ performance ratings and pay and can lower a school’s ratings, leading to state intervention and loss of students to competitors.
“There’s pressure from the superintendent and pressure from the state, and so the pressure causes some people to cave in,” said Tom Haladyna, an emeritus professor at Arizona State University and a nationally known researcher on standardized testing.
The state currently does not screen for suspicious swings in scores as it watched out for the integrity of its Arizona’s Instrument to Measure Standards test. It focuses instead on prevention by training the test givers, and it responds to complaints. In fall 2009, the state did start checking erasure marks on AIMS tests.
Gregory Cizek, a professor and testing expert at University of North Carolina at Chapel Hill, likens gains of the size posted at the 93 schools identified in the newspaper analysis to a “miracle in a bottle.”
“It could be with a small group and powerful interventions and a dedicated staff, you could pull off a miracle,” said Cizek, author of a book on detecting and preventing cheating. “But there are no currently known interventions or instructional practices that have been shown to produce gains of that magnitude.”
The data analysis used by The Republic and USA Today is similar to one used by the Florida Department of Education to identify schools with unusual gains. The Republic examined AIMS scores in Grades 3-8 at 1,337 schools over five years. It looked for groups of students, an entire grade, where the average score jumped by at least three standard deviations. Standard deviation measures variation from the average, and testing experts say a jump of three or more standard deviations is statistically improbable. The Republic found 123 instances of such jumps at 93 schools during the five years.
The schools are of various types, including neighborhood schools in large districts, schools that serve mainly poor families, and charter schools.
Principals and teachers say their schools’ extraordinary gains on AIMS are real and the result of their focus on student improvement. They also attributed gains to skilled teachers, changes in curriculum and hard work. Score drops are often caused by students moving to less experienced teachers and students moving to different schools, they said.
Consider Friendly House Academia del Pueblo, a charter school in central Phoenix. In spring 2009, mean AIMS math scores in about 25 students in the seventh grade shot from near the bottom to near the top of all of the state’s seventh-graders. The following year, they plummeted.
Principal Ximena Doyle attributed the subsequent drop to the students having a different teacher that year.
Another school with remarkable gains was Kenilworth Elementary School in central Phoenix. A strong team of seventh-grade teachers, an especially good math teacher and motivated students were responsible for significant gains in two grades, Principal Ed Flores said.
Carpe Diem Collegiate High School and Middle School in Yuma had some of the largest gains on AIMS in the state. School officials credited increases to the school’s unique learning model that combines classroom, online and video instruction, along with frequent testing to assess where students need to improve.
Huppenthal said he is not pushing to overhaul the way schools or state officials investigate or detect cheating on tests. The vast majority of schools operate with integrity, he said.
Still, “you’re fooling yourself if you don’t think there are issues out there,” he said. “So we are going to be alert and examine those issues. But I think the system they have in place has been a system that works, with a few exceptions.”