Debunking the Infamous MCAS Drop in “Proficiency”

If you’re a Beverly parent, you’re most likely familiar with the Patch article “Mayor Cahill’s Legacy: The Decline of Beverly Public Schools,” by Dr. Matthew C. Ferreira, and the infamous table that has caused consternation in our community over MCAS scores:

In the middle of the table, you see what looks like considerable decreases in percentages of ELA, Math, and Science “Proficiency,” from 2013 to 2023 with the largest decreases at 30% and 39% for Grades 3-8 MCAS ELA and High School MCAS Science, respectively. Facebook responses included angry emojis and speculation of what could cause such decreases, including COVID, lack of parental discipline, lack of teacher accountability, and overuse of social media.

As a technical editor, I edit tables of data like this every day. It is my job to ensure that data are portrayed clearly and have specific citations that correspond to inclusive reference entries so that readers can track down the data with one click. For this table in particular, we are told that the data comes from the Department of Elementary and Secondary Education (DESE), but we are not given the exact location for the data in columns 2013 and 2023. This is like finding a needle in a haystack. When I chastised Dr. Ferreira in a post on Facebook for his oversight, he said, “This is a FB post not a research paper. All the data is available for anyone to verify on DESE. Feel free to fact check.”
So that's what I did, albeit, annoyed that I had to do the work to find the data, even though it is technically the researcher’s job to provide a more exact citation so there is no question as to what the data are, and where they are, to not only effectively rule out plagiarism but also not be a lazy jerk and inconvenience the reader by imposing a scavenger hunt. After poring over pages and pages of DESE reports and data, I discovered that Dr. Ferreira had indeed used the DESE data incorrectly. I’ll show you what I mean.

First, let’s look at the facts. The 2013 MCAS test, a.k.a. “legacy test,” is very different from the 2023 or “next generation test” with different achievement levels and different scaled scoring ranges. This screenshot is from the DESE Spring 2023 MCAS Tests Summary of State Results”:

As you can see from these two tables, the “Proficient” achievement level is only valid for the obsolete 2013 legacy test and has a scaled score range of 240-258, while the 2023 next generation test has an achievement level of “Meeting Expectations” with a scaled score range of 500-529 (note: “scaled scores” are not raw data, meaning they don’t count as number of questions that are correct; they are statistically weighted raw data based on the complexity of the problem).

As the adage goes, this is like comparing apples to oranges and you cannot relate the percentage values of the legacy test “Proficient” level to the next generation test’s “Meeting Expectations.” But this is somewhat hard to accept, because percentages are so widely applicable and are so often used to normalize data, so let’s look at a specific case comparing the legacy 2013 with the next generation 2023 (colored text is a hyperlink to that score:

2013 MCAS Grade 10 ELA results (Proficiency level or higher)
State 91%
Beverly 91%

2023 MCAS Grade 10 ELA results (Meeting Expectations or higher)
State 58%
Beverly 58%

By including the state value, we can see that Beverly’s students are in line with the rest of Massachusetts and have not been in freefall from 2013 to 2023, as Dr. Ferreira would have you believe.

Thinking about this another way, we can use the analogy of the Fahrenheit and Celsius scales for temperature: water boils at 212 degrees Fahrenheit, for Celsius, the boiling point temperature is 100 degrees. If you look strictly at the two numbers 212 and 100, it looks like a drop of 112 degrees, but in reality, the two temperatures are measuring the same amount of heat. Similarly, what may look like a drop in student ability could very well be measuring the same ability level.

Overall, the district is making typical progress with respect to MCAS, according to the DESE, with a score of 53 in 2024, and well within the average growth range of 40 to 60.

The table that Dr. Ferreira posted is biased and an overall abomination with respect to MCAS scores for Beverly. To me, this misuse of DESE data warrants consequence, because it is a violation of DESE’s intellectual property (meaning Ferreira is using the data as they weren't intended to be used), but maybe that’s being too severe. At the very least, Dr. Ferreira, an Ed.D, should’ve known that you can’t link the two tests, given his advanced degree in this very subject matter.

This should also be a warning that you should take what you read in Patch with a grain of salt; there are no fact checkers there to ensure that debacles like this one do not happen.