Back in November I wrote a post describing my concerns with the way BPS was measuring quality. BPS has changed the way they're doing this, so I wanted to post an update.

My main concern with the "quadrant analysis" that BPS was using was that it only looked at whether a score was above average or below average. It didn't see any difference between a school with exceptional scores and one with slightly above average scores. On the flip side, it saw a big difference between a school that was slightly above average and a school that was slightly below average.

Of course, the fact that BPS is using only MCAS scores is a problem, too, but that's not the focus of this post. My main concern was that the measurements that were going into the BPS analysis were getting distorted by the way they were boiling them down to a single number. The good news is that they have changed this. The new analysis by MIT uses a a new MCAS measurement. This uses the same eight measurements that went into the original number, but with two changes. The first is that instead of giving equal weight to achievement and student growth, the new measurement gives twice as much weight to achievement. It's very hard to argue one way or another on this change, but it does emphasize absolute achievement (the percentage of students scoring proficient or advanced) over student growth (how much did each student improve over last year relative to other kids in the state).

The other change is that instead of looking at whether each item measured is above or below average, BPS is now applying the weighting mentioned above and then averaging the numbers. This creates a continuous measurement without arbitrary cutoffs. This will do a much better job of ranking schools based on these scores.

My other concern was that once they determined the MCAS rating, BPS was setting an arbitrary cut-off to determine whether a school is a quality or non-quality school. This is still happening. However, I do think there are some improvements here. First, BPS is now considering 1/2 of schools to be quality rather than 2/3. This may still be too many, but I think it comes much closer to capturing reality. The other is that the home-based plans are breaking the schools into four tiers. While these tiers are still fairly arbitrary, having more of them gives a better representation of the true range of quality among schools.