Using Individual Participant Data Meta-analysis to Evaluate Educational Interventions on Pupils Eligible for Free School Meals

Educational interventions provide students with the support needed to develop the skills being taught by the educational system to address functional skills, academic, reasoning, and social skills. In England, a Free School Meal (FSM) is a statutory benefit available to school-aged children from families who receive other qualifying benefits and who have been through the relevant registration process. Recently, the importance of school meals was also reminded by a famous footballer Marcus Rashford who called on ministers to offer a guaranteed “meal a day” to pupils of struggling families.  It is well known in England and around the world that children growing up in poorer families emerge from school with substantially lower levels of educational attainment. Given the importance of educational qualifications in later life, various research shows that most of these children are also likely to be disadvantaged in terms of employability, health, and wellbeing.

An individual participant data (IPD) meta-analysis method offers a more flexible and pragmatic way to synthesise evidence from existing interventions (Kontopantelis, 2018). It is a more powerful approach than relying on results of a single study approach because of its ability to pool information across multiple trials, while also accounting for the different sources of variation (Debray et al., 2015) and fully exploits the available data of individual participants without having to perform additional transition steps.  Additionally, IPD meta-analysis helps to counteract the risk that individual studies may be underpowered due to the smaller sample size of FSM pupils. Several educational interventions have improved the educational attainment on average of the pupils with socioeconomically disadvantaged background as evident from several randomised control trial (RCT) studies conducted in education, but summarising this evidence using robust IPD meta-analysis has not yet been undertaken for educational interventions.

We are working on a research paper summarising evidence from educational interventions during 2011 – 2019 funded by Education Endowment Foundation (EEF) to identify whether socioeconomically deprived children, measured through children eligible for FSM, have benefitted from educational interventions compared with peer counterparts. Data from 88 EEF educational trials with over a half-a-million pupils were included in this research. Two major groups of variables (the age of pupils or pupils “key stage” in England) and types of interventions (one-to-one, small group, whole class or whole school) were considered for meta-analysis.

We (the whole team- Bilal Ashraf, Akansha Singh, Germaine Uwimpuhwe, Steve Higgins and Adetayo Kasim) have been using a simplified individual-participant Bayesian meta-analysis model to synthesise overall impact of education interventions on attainment (and gap) in literacy and mathematics performance between subgroups of pupils.  A significant challenge has been to adequately account for heterogeneity between and within the randomised control trials. Variability between trials due to different participating populations, different outcomes with respect to scale or underlying constructs, difference in methods of how the effect size were calculated, and differences in quality of the trials plays a significant role in estimating pooled effects across trials (Brookes et al., 2001).

There is a consensus that variable measures of intervention effects are likely to produce unreliable evidence of the average effects of the interventions across trials, although some of the variability between trials can be accounted for in a random effects meta-analysis. The level of variability between trials is particularly important in IPD meta-analysis because the data will be analysed on the original scales which are likely to be different between trials. An important example in EEF trials is with respect to the different key stage results. It is also well known that schools and pupils participating in educational trials are rarely representative of the wider population of schools and pupils (Weiss et al., 2017).

While the detailed results will soon come out in our pre-print publication (we’ll add the link once it’s available), we can give you a preview here to report that EEF interventions had beneficial impacts on the literacy performance of pupils eligible for FSM, compared to maths performance which showed no overall effect. However, it is interesting to note that the pooled attainment gap (of trial effect sizes) between FSM and non-FSM was very small. While the FSM pupils benefitted more than non-FSM pupils for literacy, but not for maths performance across Key Stages. The attainment gap estimates for pupils in Key Stage 3 outcomes were positive for both maths and literacy performance. Positive attainment gap here indicates that the EEF interventions had benefitted FSM pupils in Key Stage 3 more than non-FSM pupils.  But interestingly, we found that the structure of learning environment plays a role here: individual or small-group interventions improved literacy performance of FSM pupils considerably while intervention on the whole class or school were beneficial for the maths performance.

Overall, evidence from this study can be used to identify, test and scale successful educational interventions with positive impact which can be implemented in schools to improve educational attainment of FSM children. We hope this project provides a better understanding of the different interventions’ effects, inform decisions about specific interventions to target FSM subgroups, and can be used to suggest ways to improve the design, rescale, restructure or implementation of the tested interventions among FSM children.

This blog is written by Bilal Ashraf, a statistician who is a Durham Research Methods Centre Fellow and a post-doc in the Department of Anthropology, Durham University.

This blog expresses the author’s views and interpretation of comments made during the conversation. Any errors or omissions are the authors.