Midline Report for the Mixed-Methods Cluster-Randomized Controlled Trial of Impact Network’s eSchool 360 Model in Rural Zambia

Paula Dias, AIR

Gelson Tembo, Palm Associates

Victoria Rothbard and Anaïs Toungui, AIR

Low- and middle-income countries have made significant progress toward placing children into schools, but student learning and achievement are often dreadfully low. Zambia faces many common educational challenges, such as low literacy rates and public spending that is low compared to other countries in the region.

The eSchool 360 model implemented by Impact Network (IN) represents a promising approach to improving educational outcomes by incorporating three potentially high-impact components that could create important synergies: a standardized e-learning curriculum and pedagogy, ongoing teacher training and professional development, and community ownership. AIR has designed and is implementing a mixed-methods cluster-randomized controlled trial (cluster-RCT) to determine the effects of IN’s eSchool 360 model.

About the Midline Report

Read a research summary of the midline report (PDF).

This report presents the midline results of the cluster-RCT used to determine the impact of IN’s eSchool 360 model. It examines the impact of the program on early grade reading assessment (EGRA), early grade math assessment (EGMA), Zambian achievement test (ZAT), and Oral Vocabulary assessments one year after the start of the program.

In addition, it analyzes the impact of the program on intermediate outcomes, including school attendance and enrollment, perceptions of school and education quality, aspirations about child’s education and marriage, child development, food security, and education expenditures.

We present two main types of effects. The first, called the Intention-to-Treat effect (ITT), considers the impact on all children in close proximity of the schools, regardless of whether they attended the school. The ITT identifies the impact of the opportunity to attend an IN school. The second, called the Treatment Effect on the Treated (TOT), estimates the effect for those children who attend an IN school when given that opportunity.

Midline Results

Midline results suggest that the opportunity to attend an Impact Network (IN) school positively affected children’s learning outcomes across the board.

View this webinar recording that describes the study and its implications for technology-aided instruction in sub-Saharan Africa:

Effects on Test Scores

On average, we found statistically significant effects on scores 14 months after the start of the program:

0.40 standard deviations or 3.5 percentage points on EGRA scores;
0.22 standard deviations or 4.9 percentage points on EGMA scores;
0.16 standard deviations or 3.1 percentage points on ZAT scores; and
0.25 standard deviations or 6.0 percentage points on Oral Vocabulary scores.

Treatment Effects

TOT effects were substantially larger than ITT effects. On average, for students who reported that they attended IN schools three times in the last week, participating in the eSchool 360 model resulted in improvements of:

0.83 standard deviations or 7.2 percentage points in EGRA scores;
0.45 standard deviations or 10.1 percentage points in EGMA scores;
0.32 standard deviations or 6.3 percentage points in ZAT scores; and
0.52 standard deviations or 12.4 percentage points in Oral Vocabulary scores.

For students who reported that they had ever enrolled in IN schools (over the past year), we found TOT effects of:

0.68 standard deviations or 5.8 percentage points in EGRA scores;
0.37 standard deviations or 8.2 percentage points in EGMA scores;
0.26 standard deviations or 5.2 percentage points in ZAT scores; and
0.42 standard deviations or 10.1 percentage points in Oral Vocabulary scores.

Mixed-methods evidence suggested the positive effects were primarily driven by improvements in the quality of education, increases in school attendance for both teachers and children, and strong fidelity of program implementation.

Despite the positive program effects, children residing in IN catchment areas scored an average of only 11% correct on EGRA assessments and 24% correct on EGMA assessments.

Next Steps

The results of the longer-term impact and cost-effectiveness analysis will show evidence on whether and how the eSchool 360 model can be moved to scale. We will collect cost data of the eSchool 360 model in 2020. In addition, we will estimate longer-term impacts following the endline survey three years after the start of the program. We will present these results in a future endline report.