EAC Meeting Review - 12/06/2012
Addthis
The EAC met last night to hear a presentation on the progress made by Peng Shi and others towards creating a demand model among other items. Here are the major items discussed.
Data validation
Laura Perille, co-chair of the EAC data sub-committee, presented a brief report on efforts to validate the data that BPS and the EAC have been using. This evaluation is being led by a professor at Babson who specializes in data modeling and data quality management. So far, his team has validated the student and school choice data sets. They have also validated the integration of data on students and schools. Work on validating data on schools is on-going.
Miren Uriarte expressed concern that collection of data by the district may be flawed in some cases. Laura said that this is not being validated and that it was probably something that could not be done by the EAC. Miren and others agreed, but she wanted to be sure that members understand the limitations of the data.
Voting Procedures
The committee reviewed a memo on voting procedures based on the discussion from the previous meeting. Miren Uriarte asked for a “none of the above” option for those who don’t agree with any model being voted on. I had suggested this as a way to prevent a scenario where the EAC would recommend a plan that was actually opposed by a majority of its members. Craig Lankhorst said that it's common to use "present" as a voting option. Those voting present would be counted when determining whether a majority is reached while those abstaining are considered to have not voted. Miren agreed that this would address her concerns.
Bill Walczak and John Nucci voiced opposition to allowing members to vote "present." Mary Tamer also seemed to be opposed. They don’t want full year’s work to result in no recommendation. Laura Perille, Ruthzee Louijeune, Carolyn Kain, and Helen Dájer all spoke in favor of allowing a "present" vote. The committee voted 13 to 5 in favor of adding an option to vote present.
I thought this was all well and good, but last night it occurred to me that there is still a flaw in the new procedure. While the original proposal could have allowed the EAC to recommend a plan that a majority opposed, the new procedure could prevent the EAC from recommending a plan that the majority supports. Consider a scenario where there are two plans that most members like, but disagreement about which is the better of the two. When the committee votes on these two plans, 11 vote for plan A, 9 for plan B, and 5 vote "present." No plan would be recommended because only 11 of 25 voted for any plan. The problem is that some or all of the 9 who voted for plan B might vote for plan A if given a chance to vote yes or no.
I will propose to the committee that instead of eliminating plans until they are voting on two plans that they actually eliminate options until they have one plan. Then they would vote yes or no on whether to recommend the final plan. There wouldn't be any reason to allow a present vote for the final recommendation, but it could be allowed. I would suggest that the committee do this even if a majority is achieved without eliminating all the other plans. This will allow members to rally behind the final recommendation if they wish to.
Presentation by Peng Shi
Peng then presented on the progress being made by the team at MIT. The presentation is titled "Progress Update: A Holistic Understanding of 'Access to Quality'." He started by saying, "If the centerpiece of this process is access to quality we want to make sure we get this right." He pointed out that while everyone has a different definition of quality, some factors measure school quality, while others are really about a quality match and vary from family to family. Measures of school quality can include MCAS scores, teacher and principal effectiveness, facilities, etc. Measures of a good match can be distance from a child's home, program match, philosophy, etc. In summary, "a quality choice is not only a quality school, but a quality match for my family."
I won't go through the entire presentation as it will probably be on-line soon, but I will try to cover some major points. Peng showed some examples of how families in different neighborhoods choose schools. Families definitely prefer nearby schools, but in some areas more distant schools are more popular. More analysis is needed, but what we might expect appears to be true, families want their children to go to school close to home, but are willing to send them out of the neighborhood when they don't believe the nearby schools are of sufficient quality.
As he works to create a demand model he will try to quantify how families are using distance and MCAS scores to choose schools. He needs to answer questions like how much more likely is a family to choose a school if its MCAS scores are 10% higher? If it's one mile closer to home?
Furthermore, a demand model can be used to measure quality as families define it. While we may not know why families are choosing one school over another, the fact that they are can indicate that they consider some schools to be higher quality than others. There was a fair amount of discussion on how reliable this kind of measurement can be. It assumes that parents are making informed choices and that they are selecting based on their actual preference and not employing a strategy to try to get the best results. While the algorithm currently being used by BPS is considered "strategy proof" (meaning that parents get the best results by expressing their true preferences), there is a potential flaw. Because children can only be assigned to up to three wait lists, some parents will choose a less popular school as one of their top three choices to increase their chances of getting on a wait list with a reasonable chance of getting in.
Also, some parents might not understand the algorithm and may choose to some extent based on their perceived chance of getting a seat at a given school. One parent after the meeting suggested that some families who are choosing walk zone schools are choosing them for the increased chance of getting a seat rather than an actual preference. Peng can adjust for that by looking at whether walk zone influences choices independent of distance. Peng and I spoke about this after the meeting and I suggested that he compare families' top two choices to the entire list of choices to see if there's a difference in how they choose. I think this will show us if the wait list issue is a problem because most families would put a less-desired school as a third choice if they are concerned about this. Peng said he can do that.
As for the concern about families who are less well-informed, Peng can break out how families of English-language learners choose vs. the entire population. He can do the same for low-income families. That will help see if these families are making different choices.
Peng also proposed some ways to measure how good children's access to quality would be under any plan. He suggests looking at:
- What percentage of families would have access to their top choice school.
- What chance would families have to get into a school they considered "not bad." Some baseline of minimum acceptable quality would have to be established, but this would boil down to chances of getting into at least a "middle-of-the-road" school.
- Chance of being assigned to any school in the list of available choices. This is really a measure of how well the plan is managing capacity. How many children would get no assignment and have to wait for a space to open?
The appendix of Peng's presentation had many interesting statistics on school choice. I highly recommend reading it.
After the presentation there were several questions from the EAC and some discussion. Ian Deason wanted to know if populations like English-Language learners and low-income families can be looked at separately. Kathleen Colby raised the issue of people not expressing their true preferences because of the three wait list limit.
Miren Uriarte asked if they can actually develop a better plan from the demand model? Peng says they will try, but can't make promises. My understanding is that you may be able to do this somewhat through trial and error. Make a change to a plan and see what happens. If it improves results, continue in that direction until the optimal result is achieved.
There was also some discussion about socioeconomic balance. Mary Tamer feels that shouldn't be part of the discussion and cited research that low-income children don't benefit until around 60% of students are middle class. Since less than 30% are middle class in BPS, that can't be achieved. Others, including Craig Lankhorst and Ron Gittens felt it was important.
Public Comment
John Mudd said that if the committee is considering changing the way walk zone works they need to have Peng analyze that. Barbara Martinez presented over 1,200 signatures from people asking for sibling grandfathering. Another parent said that the analysis will need to tease out the difference between families who make informed choices and families who don't. Ann Walsh asked the committee to choose something that will prevent so many students from being unassigned. Anna Ross (who wrote this op-ed) spoke about importance of sibling preference and community as a factor.
Next Steps
BPS had been planning to present information on details of the current algorithm and on the effects of grandfathering at this meeting, but didn't feel that there was enough time. They asked the EAC to set aside time in the future for this. An additional meeting could be scheduled in December or January, but no decision was made.
The next EAC meeting is Thursday, 12/13 at Suffolk University. I believe it will be at 73 Tremont St. where the committee has met before. This meeting will be to go through the committee's draft recommendations line by line.
Comments
Implementation issues
Josh, very nice job of summarizing the discussion. Even having been there for most of it, it's useful to re-read.
Next week will be interesting, one of the more substantive meetings in a while, as the discussion will shift from the details of the plans to the recommendations.
A number of the issues that have been raised are really implementation issues. Laura Perille suggested a few meetings back that they start to collect some of those, that was a good suggestion that AFAIK hasn't been mentioned since. By noting them in the EAC's recommendations, there's a better chance something might get done, and it may help set priorities. For example, fixing the waitlist is to me more an implementation issue, as is improving the current efforts to help families make the right choices (either decoding the process, or crossing a language gap).
I do wish they'd stop worrying about whether this is "strategy proof" enough and start fixing the glaring problems. Such as the waiting list problem - can't parents just pick three of their top five after the fact, or some similar fix? And just plain communication - as you know, the current system is very poorly understood - a lot of legends and old info gets passed along from parent to parent. All compounded when language barriers come into play.
I think the choice of vocabulary in Peng's presentation may be confusing, especially for those reading it outside the meeting. Access to quality has meant something else. Is it being redefined, or is he just explaining how a demand model works? My take on it is, there's quality (measured independent of a particular family's preferences), and there's match, and both determine how a family chooses.
Overall, though, giving the EAC more time for analysis, and the EAC's committment to actually do the analysis, has the potential to build the best possible plan (by a wide margin), and to build a great foundation for the future. The tools can evolve and improve over time in a way that none of the zone plans can.
The best thing about that is not having to do this again anytime soon, and being able to focus on the real issue - continuing to improve quality in the BPS.