The Shanker Blog wrote about our Performance Management Framework. Here's their post:
For elementary and middle schools, the DC Performance Management Framework (PMF) is a weighted index composed of: 40 percent absolute performance; 40 percent growth; and 20 percent what they call “leading indicators”. The index scores are then sorted into one of three tiers, with Tier 1 being the highest, and Tier 3 the lowest.
So, these particular ratings weight absolute performance – i.e., how highly students score on tests – a bit less heavily than do most states that have devised their own systems, and they grant slightly more importance to growth and alternative measures. We might therefore expect to find a somewhat weaker relationship between PMF scores and student characteristics such as free/reduced price lunch eligibility (FRL), as these charters are judged less predominantly on the students they serve. Let’s take a quick look.
The most simple descriptive statistic is to simply calculate the average FRL and special education rates of the schools within each tier (again, charters are sorted into Tiers 1-3, with Tier 1 being the highest performing, at least according to this system). These averages are presented in the table below for all 51 elementary/middle charter schools that received a 2012 rating (keep in mind that this is a small group of schools – 51 in total, and that most are concentrated in Tier 2, which means that the averages can be influenced substantially by a small set of schools).
There is no apparent relationship between tier and special education (the rightmost column) – schools in Tier 1 actually have the highest rates.
The FRL averages, in contrast, do exhibit a pattern – schools in the lower tiers have a higher FRL rate, on average, than schools in the higher tiers. The difference is most stark when comparing Tier 1 (66.4 percent) to Tier 2 (77.7 percent). This is to no small extent a result of the measures employed by the system – not only of the proficient/advanced rates, which are heavily associated with characteristics such as FRL, but also the growth measure (median growth percentiles), which usually are modestly correlated with these traits.
Averages, of course, sometimes mask what’s happening underneath, so it might be useful to take a look at the scatterplot of FRL by the actual index scores upon which the tier ratings are based.
You can see a relationship, but it is quite messy. Although virtually all of the schools in Tier 3 (those with index scores below 35 percent) are high poverty, the schools with the four highest index scores have FRL rates over 80 percent, while several schools with rates below 50 percent receive relatively low index scores.***
It is therefore fair to characterize this relationship as discernible (statistically and otherwise) and modest-to-low. It is also noticeably less strong than that in most of the other states I have reviewed.
The reason for this is very simple: The PMFs rely less heavily on absolute performance measures, which are strongly correlated with student characteristics such as FRL. This does not necessarily mean these DC charter ratings are “better” or more fair than their counterparts elsewhere, but it does show how the choice of constituent measures has a predictable and often substantial impact on the association between the results and the students schools serve.
Do you agree?