Is Progress 8 a valid and reliable measure of school effectiveness?

Abstract

Policy-makers, school leaders, parents and citizens want to know whether schools are doing their job well, and whether particular schools or types of schools are doing that job particularly well. Insofar as the job is defined in terms of pupil attainment in public examinations, value added models are currently preferred. However, both the validity and reliability of value added models have been questioned and the debates about their fairness remain unresolved. One of the major problems for value added models is that while raw-scores for each school are reasonably stable over time the value added scores based on them are more volatile. This instability does not prove that there is a problem with the measures, but it is how construct irrelevant variance would manifest. This thesis addressed these concerns by scrutinising the validity of Progress 8, the Department for Education’s headline indicator of school performance in England. More specifically, it investigates whether the differences between schools’ annual performance ratings and the change in schools’ ratings over time can be explained by the kinds of factors that educational effectiveness is usually attributed to and perhaps more importantly, whether these factors are under the control of schools. The results show two things. First, that the Progress 8 scores are biased by external variables such as the differences in schools’ intake and examination entries. This is profoundly unfair and is likely to mean that the wrong schools are identified as differentially effective. And second, that even school leaders with expert knowledge of their institutions, access to students’ performance data and the previous year’s attainment averages cannot make reliable predictions about schools’ value-added results. This outcome invalidates the notion that parents can use Progress 8 outputs as a means of making informed decisions about the effect of their child attending one school over another.