Eric.ed.gov – Repetita Iuvant? Lessons from Repeated RCTs on the Effectiveness of a Teacher Professional Development Program

This work summarizes the results of two randomized control trails (RCTs) aimed at evaluating the effectiveness of a professional development program for lower secondary school math teachers. The program, called M@t.abel, was financed by the Ministry of Education in Southern Italy with EU funds. It lasts a full school year and it is based on formal and on-line tutoring, providing alternative methods for teaching traditional math contents. The program and its evaluation were both held at scale, a situation far from artificial settings and allowing for the observation of real constraints faced in delivering such a service. This situation, along with the replication of the RCT on two consecutive cohorts of teachers, provides several opportunities and challenges both from the policy and research practice points of view. The main outcome variable is student math achievement, but effects of the program on student attitudes are also explored (e.g., attitudes towards math, the motivation to study, the perception of the curriculum pace and text anxiety, etc.). A big effort in the collection of monitoring data allows the authors to better understand how the program works. Changes in the effectiveness of the program were detected from the first to second RCT. They are interpreted in the light of changes in some features of the program. In this evaluation, researchers managed to engage with policymakers and practitioners in the planning stages of the program and before even starting, the evaluation contributed to program design (streamlining the treatment, focusing on contents and ensuring collaboration among the different institutions). Randomization through delayed treatment resulted acceptable under technical and ethical profiles and response rates were high from both treatment and control groups, and from both teachers and students. The repeated RCTs bearing different results made the task for researchers to communicate results particularly difficult and requested an increase effort in collecting and analyzing implementation data. One table is appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]