Eric.ed.gov – Evaluating Phase II of a New York City-Wide STEM Initiative Using Propensity Score Methods: A Replication Study

Recently, the authors have been exploring the use of propensity score methods for developing evidence of program impact. Specifically, they have been developing evidence (after one year of implementation) of the effects of the Math Science Partnership in New York City (“MSPinNYC2”) on high school students’ achievement–both in terms of course grades and scores on end-ofcourse tests in two key Science, Technology, Engineering and Mathematics (STEM) disciplines: Integrated Algebra and Living Environment. Using an evidence-based approach which relies on propensity score matching, the authors asked if the program in its early stages is making a difference in students’ academic achievement and college readiness. The “MSPinNYC2” program restructures early high school STEM courses to include 6-8 Teaching Assistant Scholars (TAS) who, along with the teachers, facilitate in-classroom group work on a daily basis. Early pilot studies suggested the model of peer-enabled restructured classrooms (PERC) increases student achievement and narrows the achievement gap in high school STEM courses. In its inaugural year the “MSPinNYC2” project recruited over 700 students (n = 711) in four New York City public high schools to participate in PERC classes. The students served by the project are not the academically elite (e.g., only about 20-25% were proficient in math and/or English language arts at the end of 8th grade). In the first year (2011-12) the PERC courses were taught by eleven different high school teachers–all trained in the PERC pedagogical model by the Program’s staff. Each course served between 25-30 students who were tutored in-class by 6 or 7 TAS. The “MSPinNYC2” program is a multi-year, multi-site STEM intervention, with new cohorts of students entering the program each year, it is essential to develop a sound, rigorous method for evaluating the effectiveness and the scalability of the intervention–one that can be replicated each year. The authors are keenly interested in models that allow them to replicate their results from year-to-year with each new cohort of students. Thus, the purpose of this proposed study is to explore the use of a propensity score model that uses a genetic matching algorithm to evaluate the replicability of limited first-year findings, using as participants a new cohort of students who participated in the second year of the program. Specifically, they examine: (1) the effectiveness of using propensity score methods to evaluate a math/science educational intervention; and (2) the replicability of the findings on a new cohort of students. The current study uses an observational design and is a replication of the initial, preliminary propensity score analysis conducted at the end of the first year of the program. The authors describe the variables used, and will continue to use, during the replication phase of the program evaluation efforts. The results from Phase I of the study are presented, while Phase II is still underway during the production of this paper.