Eric.ed.gov – Locating Differential Effectiveness of a STEM Initiative through Exploration of Moderators

In randomized control trials, such as the one the authors conducted in Alabama, moderators are potentially an important source of variation in treatment effects. Whether moderators used in the RCT’s analysis are pre-existing characteristics of students, their teachers, or the school or other organizational context, they can point toward differences in the way the program (treatment) is implemented, the way teachers interact with different kinds of students when implementing the program, or how effects are operating at levels in the organization above the level where the differences were measured. The authors appreciate Cronbach’s (1975) metaphor of the “hall of mirrors” to point to the endless possibilities for higher-order interactions complicating the explanation of what was found. But the goal of this paper is to illustrate a disciplined approach to exploring important interactions guided by the authors’ interest in narrowing the working hypotheses as to how the program is having its effects and thus guiding practitioners, program developers, and policy makers in making the most effective use of STEM programs. The Alabama Math Science and Technology Initiative (AMSTI) is a major program developed and deployed in Alabama schools. Partly motivated by the 1996 National Assessment of Educational Progress scores, which were below the national average for Alabama’s grade 4-8 students in mathematics and grade 8 students in science, the Alabama State Department of Education (ALSDE) developed a statewide initiative to improve mathematics and science teaching and student achievement in K-12 schools. AMSTI, a school-wide intervention, was introduced in a set of 20 schools in 2002. Each year since then, the state has rolled out the program to additional schools within its 11 regions. By 2009, about 40 percent of the state’s 1,518 schools were designated as AMSTI schools. The RCT measured the impact of AMSTI. Key outcomes included student achievement in mathematics problem solving, science, and reading; the use of active learning instructional strategies in mathematics and science classrooms; and teacher content knowledge and student engagement in mathematics and science.