MCAS scores don't provide a meaningful comparison of schools

 

By DANIEL E. GOREN, Standard-Times staff writer

NEW BEDFORD -- Unseen in the debate over whether charter schools perform better than regular public schools is a deeper story -- one hidden in tricky percentages and perhaps not best told by scores on a single standardized test, experts say.

At first glance, the city's charter school, called New Bedford Global Learning, is not doing as well as most of the city's regular public schools in sixth-grade math. But it is doing slightly better in seventh-grade English and eighth-grade math, according to 2004 test results from the Massachusetts Comprehensive Assessment System.

"The problem is that the assumption that meaningful comparisons can be made is quite shaky," said Anne Wheelock, an education policy analyst for Boston College. "Without more information -- more context -- comparisons don't have much meaning."

MCAS scores are traditionally shown in percentages of students in four categories -- advanced, proficient, needs improvement and "warning" or failing. This leaves the average person believing a "rational" comparison can be made between charter and regular public schools based on those scores, Ms. Wheelock said.

And with lots of money at stake and each student representing a price tag, those comparisons, often missing vital information, can make school administrators squirm.

Global Learning is young, having only taken the MCAS in 2003 and 2004. As a result, many say only time will tell if it is a success or failure.

"My honest perception is that I need to see more," said Michael E. Longo, superintendent of New Bedford's schools.

One of the largest factors in determining whether a school tests well on MCAS is not necessarily its teaching methods but the makeup of its student body, Ms. Wheelock said.

"Student characteristics -- poverty, disabilities, first language at home -- have a powerful bearing on test scores," she said. "For the most part, charter schools in general are enrolling fewer students in poverty, fewer students with disabilities, and fewer students who are learning English as a second language than schools in comparable districts. This makes it difficult to compare MCAS scores."

In New Bedford's regular schools, 62 percent of the students come from low-income families, 18 percent are in special education, and 22 percent speak limited English or do not speak English as their first language, according to the state Department of Education. At Global Learning, 55 percent come from low-income families, 20 percent are in special education and 6 percent do not speak English as their first language.

Global Learning's small numbers of students still learning English -- 14 of 237 students, according to the state -- could give it an edge on the English MCAS test. By comparison, at regular middle schools such as Keith, Roosevelt and Normandin, 682 out of 3,074 students have trouble with English.

"Given that it's harder for students learning English to cope with English MCAS, and given that the charter school is enrolling a much lower percentage of students learning English, you'd expect the middle schools to have lower scores and the charter school to have higher scores on the English test," Ms. Wheelock said.

That is in fact the case. On the seventh grade English test, Global learning had 15 percent more of its seventh graders in the advanced and proficient categories.

One reason why charter schools might have fewer troubled, low-income or language-challenged students is that their pool of children is self-selecting, experts say. Parents must first apply for their children to attend, and more likely than not, parents with knowledge enough to choose a charter school likely already have advantages. Theoretically, this leaves New Bedford's most challenging students to teach in regular schools.

"We deal with all the kids," Mr. Longo said.

But it is not always the case that charter schools get the "cream" off the top of the student pool, said Paul Fay, principal of Global Learning.

"Selection can go in both directions," he said. "We have kids referred here because they have not had success in the regular public schools."

As Mr. Fay says is the case with Global Learning, some parents with troubled children choose to send their kids to charter schools for the smaller, more personal atmosphere, he said.

Not every test score is created equal, researchers say. The more students that are tested, the more reliable the results. The fewer tested, the less reliable the results.

The reason is that with smaller numbers of students tested -- usually less than 100 -- the outcome can be easily influenced by one exceptionally bad student or by one superstar student. If one year a superstar brings up everyone's average and the next year a bad student drags the average down, scores could vary from year to year by 15 to 20 percentage points, Ms. Wheelock said.

The school ends up "taking 'credit' or 'blame' when, in fact, there's nothing more than luck involved," she said.

Having a small sample size was the case for Global Learning, with 63 eighth graders taking the 2004 MCAS math test, 75 taking the seventh grade English and 59 taking the sixth grade math.

 

But strange things can influence standardized test scores, according to research done by Doug Staiger, a professor at Dartmouth College in New Hampshire. A dog barking in the parking lot on the day of the test, a particularly severe flu season that year and bad weather on the day of the test can all change its results, he said.

The number of students who were enrolled at the school in the beginning of the year, but who left before taking the MCAS test, can also influence the results.

Scores can change dramatically by holding children back or removing children in some way before testing occurs. This is particularly true if the students leaving the school are the troubled, or lower performing students, Ms. Wheelock said.

In the case of Keith, Roosevelt and Normandin, the percentage of students no longer enrolled before taking the MCAS was fairly small -- on average 3 percent.

But in the case of Global Learning, the percentage was slightly higher -- 5 percent of the students enrolled in seventh grade in 2004 left the school before MCAS tests were administered. Ten percent of eighth graders had also gone. That's 11 students out of the original 149 enrolled in those grades who did not make it to testing time in the spring.

If those students were the worst of the bunch, their departure before testing took place could prop up Global Learning's scores.

But in this case, Mr. Fay says a few of those who left simply moved away, and that many were good students, not bad.

"To be honest, I'd take all of them back, since they would probably have helped our scores," he said.

Ms. Wheelock agrees that someone has to keep track of how well schools are performing, and that maybe MCAS has a role in that effort.

But in the end, student work such as essays, math homework, presentations and other projects might be better measures of success than MCAS test results, Ms. Wheelock said. She said she has found the test to be a "very poor yardstick for assessing school performance."

Even worse, she said the scores can be misleading and create pointless ill will between charter schools and regular public schools that have the same goal: education.

"These scores just don't tell the story of the extent and quality of learning that's going on in any school," she said.

This story appeared on Page A1 of The Standard-Times on February 6, 2005.