Educational Evaluation and Policy Analysis recent issues

Can Interdistrict Choice Boost Student Achievement? The Case of Connecticut's Interdistrict Magnet School Program

Bifulco, R., Cobb, C. D., Bell, C. — Fri, 18 Dec 2009 10:15:37 PST

Connecticut’s interdistrict magnet schools offer a model of choice-based desegregation that appears to satisfy current legal constraints. This study presents evidence that interdistrict magnet schools have provided students from Connecticut’s central cities access to less racially and economically isolated educational environments and estimates the impact of attending a magnet school on student achievement. To address potential selection biases, the analyses exploit the random assignment that results from lottery-based admissions for a small set of schools, as well as value-added and fixed-effect estimators that rely on pre–magnet school measures of student achievement to obtain effect estimates for a broader set of interdistrict magnet schools. Results indicate that attendance at an interdistrict magnet high school has positive effects on the math and reading achievement of central city students and that interdistrict magnet middle schools have positive effects on reading achievement.

Student Participation and Performance on Advanced Placement Exams: Do State-Sponsored Incentives Make a Difference?

Jeong, D. W. — Fri, 18 Dec 2009 10:15:37 PST

Many states provide incentives to students, teachers, and schools for the participation and success of students on Advanced Placement (AP) examinations administered by the College Board. The purpose of this article is to examine whether these incentives help students enroll and succeed in AP exams. An analysis of nationally representative AP exam data, taken from the Education Longitudinal Study of 2002, revealed that AP exam fee exemption, the most prevalent incentives, leads to an increase in the likelihood of AP course enrollees taking the exam—in particular, the disadvantaged. In contrast, little evidence was found that performance-based incentives, to which several states link AP test results, are helpful for improving AP exam participation and performance.

College Preparatory Curriculum for All: Academic Consequences of Requiring Algebra and English I for Ninth Graders in Chicago

Allensworth, E., Nomi, T., Montgomery, N., Lee, V. E. — Fri, 18 Dec 2009 10:15:37 PST

There is a national movement to universalize the high school curriculum so that all students graduate prepared for college. The present work evaluates a policy in Chicago that ended remedial classes and mandated college preparatory course work for all students. Based on an interrupted time-series cohort design with multiple comparisons, this study found that the policy reduced inequities in ninth grade course work by entering ability, race/ethnicity, and special education status. Although more students completed ninth grade with credits in algebra and English I, failure rates increased, grades slightly declined, test scores did not improve, and students were no more likely to enter college. In sum, few benefits resulted from universalizing college preparatory course work among freshmen, but dropout rates did not increase. Possible explanations are discussed.

Excused Versus Unexcused: How Student Absences in Elementary School Affect Academic Achievement

Gottfried, M. A. — Fri, 18 Dec 2009 10:15:37 PST

The literature on school absences has focused predominantly on the reasons for student truancy, or it has assessed only aggregate student absences in their effect on achievement. However, this study brings forth a new issue: the relationship between types of absences—excused versus unexcused—and school performance. With a quantitative model of educational achievement on a longitudinal multilevel data set of all second-through fourth-grade students in the Philadelphia School District from 1994 to 2000, this study disaggregated absence information to provide new insight on the attendance–achievement relationship. Specifically, a model using fixed effects with classroom-level clustering was employed to determine how the distinction among varying proportions of excused versus unexcused absences related to students’ standardized test performance in reading and math. This article demonstrates that distinguishing between students with high rates of excused or unexcused absences is significant. Having a higher proportion of excused absences to total absences exhibits a positive relationship between reading and math test scores. Conversely, students with a higher proportion of unexcused absences places them at academic risk, particularly in math achievement and as early as in elementary school. Implications for policy are discussed.

Teacher Preparation and Student Achievement

Boyd, D. J., Grossman, P. L., Lankford, H., Loeb, S., Wyckoff, J. — Fri, 18 Dec 2009 10:15:38 PST

There are fierce debates over the best way to prepare teachers. Some argue that easing entry into teaching is necessary to attract strong candidates, whereas others argue that investing in high quality teacher preparation is the most promising approach. Most agree, however, that we lack a strong research basis for understanding how to prepare teachers. This article is one of the first to estimate the effects of features of teachers’ preparation on teachers’ value added to student test score performance. Our results indicate variation across preparation programs in the average effectiveness of the teachers they are supplying to New York City schools. In particular, preparation directly linked to practice appears to benefit teachers in their 1st year.

The Influence of On-Campus, Academic Year Undergraduate Research on STEM Ph.D. Outcomes: Evidence From the Meyerhoff Scholarship Program

Carter, F. D., Mandell, M., Maton, K. I. — Fri, 18 Dec 2009 10:15:38 PST

The Meyerhoff Scholarship Program, which celebrated its 20th year in 2008, is considered a successful intervention program for increasing the number of underrepresented minorities who earn Ph.D.s or M.D./Ph.D.s and pursue research careers in science, technology, engineering, and mathematics (STEM). This article examines the relationship between participation in one specific component of the Meyerhoff Scholarship Program—on-campus, academic year research—and the pursuit of a STEM Ph.D. by 13 cohorts of program participants. The results indicate that participation in on-campus, academic year research is associated with a substantial increase in the probability of pursuing a STEM Ph.D. They further suggest that the structure and intensity of the on-campus, academic year research experience matter.

Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison

Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., Cook, T. D. — Fri, 18 Dec 2009 10:15:38 PST

Adjustment methods such as propensity scores and analysis of covariance are often used for estimating treatment effects in nonexperimental data. Shadish, Clark, and Steiner used a within-study comparison to test how well these adjustments work in practice. They randomly assigned participating students to a randomized or nonrandomized experiment. Treatment effects were then estimated in the experiment and compared to the adjusted nonexperimental estimates. Most of the selection bias in the nonexperiment was reduced. The present study replicates the study of Shadish et al. despite some differences in design and in the size and direction of the initial bias. The results show that the selection of covariates matters considerably for bias reduction in nonexperiments but that the choice of analysis matters less.

Quality of Research Design Moderates Effects of Grade Retention on Achievement: A Meta-Analytic, Multilevel Analysis

Allen, C. S., Chen, Q., Willson, V. L., Hughes, J. N. — Fri, 18 Dec 2009 10:15:38 PST

The present meta-analysis examines the effect of grade retention on academic outcomes and investigates systemic sources of variability in effect sizes. Using multilevel modeling (MLM), the authors investigate characteristics of 207 effect sizes across 22 studies published between 1990 and 2007 at two levels: the study (between) and individual (within) levels. Design quality is a study-level variable. Individual-level variables are median grade retained and median number of years postretention. Quality of design is associated with less negative effects. Years postretention is negatively associated with retention effects, and this effect is stronger for studies using grade comparisons versus age comparisons. The results challenge the widely held view that retention has a negative impact on achievement. Suggestions for future research are discussed.

The Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education

Slavin, R., Smith, D. — Fri, 18 Dec 2009 10:15:38 PST

Research in fields other than education has found that studies with small sample sizes tend to have larger effect sizes than those with large samples. This article examines the relationship between sample size and effect size in education. It analyzes data from 185 studies of elementary and secondary mathematics programs that met the standards of the Best Evidence Encyclopedia. As predicted, there was a significant negative correlation between sample size and effect size. The differences in effect sizes between small and large experiments were much greater than those between randomized and matched experiments. Explanations for the effects of sample size on effect size are discussed.

Acknowledgements

Fri, 18 Dec 2009 10:15:38 PST

To Choose or not to Choose: High School Choice and Graduation in Chicago

Lauen, D. L. — Thu, 13 Aug 2009 16:17:00 PDT

School choice reforms have been proposed as ways to enhance efficiency, equity, and effectiveness in education. This study examines the consequences of participating in public high school choice in Chicago, a city with a wide variety of choice programs, including career academies, charter schools, magnet schools, and selective test-based college prep high schools. The analysis uses population-level administrative and survey data on all public school eighth graders enrolled in Chicago to estimate the effect of school choice participation on on-time graduation propensity (i.e., in 4 years). Techniques employed to estimate this effect include propensity score, catchment area fixed effects, and multilevel analysis. Results suggest that there is a modest positive graduation benefit from exercising school choice. There are no racial/ethnic differences in the choice benefit, but low-achieving students benefit less from high school choice than high-achieving students. In addition, students in high-poverty neighborhoods gain less from exercising choice than do students in low-poverty neighborhoods. These findings call into question the extent to which school choice enhances equity for low-achieving students and students in high-poverty neighborhoods.

A Longitudinal Investigation of the Relationship between Teachers' Self-Reports of Reform-Oriented Instruction and Mathematics and Science Achievement

Le, V.-N., Lockwood, J. R., Stecher, B. M., Hamilton, L. S., Martinez, J. F. — Thu, 13 Aug 2009 16:17:00 PDT

In the past two decades, several major initiatives were launched to improve mathematics and science education. One prominent feature in these efforts was a new approach to teaching mathematics and science, referred to as reform-oriented teaching. Although past studies suggest this approach may improve student achievement, the relationships between reform-oriented pedagogy and achievement were weak. The weak relationships may be partially attributable to the limited time frame in which reform-oriented teaching was examined (typically a 1-year period). This study explored the relationship between mathematics and science achievement and reform-oriented teaching over a 3-year period. Results suggested greater exposure to reform-oriented instruction was generally not significantly associated with higher student achievement but the effects became stronger with prolonged exposure to reform-oriented practices. Reform-oriented instruction showed stronger, positive relationships with open-ended measures than with multiple-choice tests in both mathematics and science and with problem-solving skills than with procedural skills in mathematics.

Do Inspections Improve Primary School Performance?

Luginbuhl, R., Webbink, D., de Wolf, I. — Thu, 13 Aug 2009 16:17:00 PDT

This article investigates the effect of Dutch primary school inspections on test scores. These inspections are assessments of the educational quality of schools. The authors use two approaches to estimate the effect of the inspections. The first approach, based on a standard fixed effects model, indicates that inspections lead to slightly better school performance: Test scores increase by 2% to 3% of a standard deviation. However, the fixed effects model might be sensitive to selection bias. The second approach exploits a sample of randomly selected schools originally drawn for the purpose of compiling the annual report of the state of Dutch education. Based on this approach, the authors find that inspections do no harm but seem to have little or no effect on student performance.

Is There a de Facto National Intended Curriculum? Evidence From State Content Standards

Porter, A. C., Polikoff, M. S., Smithson, J. — Thu, 13 Aug 2009 16:17:00 PDT

State content standards are the backbone of the standards-based reform movement. Content standards provide teachers with a set of guidelines for what students are expected to know and be able to do, defining the intended curriculum. And although the current 50-state system of education gives each state the task of setting content standards, there has been little empirical investigation of the similarities and differences among state content standards. This analysis uses the content analysis procedures of the Council of Chief State School Officers/State Collaboratives on Assessment and State Standards to consider whether there exists a de facto national curriculum as defined in state content standards. Data from English/language arts and reading (ELAR), science, and mathematics for Grades 4, 8, and K–8 are used. Results suggest considerable variability among states in the content of content standards, particularly in individual grades, but also for the aggregated standards. Further analysis suggests that state standards are no more well aligned to national professional standards (i.e., National Science Education, National Council of Teachers of Mathematics) than to other states’ standards. Still, there exists a small "core curriculum" across states in each content area. The level of focus of the state standards varies substantially across states, and the redundancy of the standards is such that alignment within state, across grades is often as high as alignment within grade, across states. Policy implications are briefly discussed.

How External Interveners Leverage Large-Scale Change: The Case of America's Choice, 1998-2003

Glazer, J. L. — Thu, 13 Aug 2009 16:17:00 PDT

This article discusses a dilemma that confronts reformers who seek to improve the education system yet operate from a position outside that system. On one hand, "external" reformers have found that the implementation of school-level programs is attenuated by a lack of coherence and support in schools' environment. Districts and states often enact policies and pursue initiatives that are not consistent with the aims and practices of intervention designs. On the other hand, when reformers redraw the boundaries of their operations to include districts and states, they are beset with an array of new problems inherent in such partnerships. The current discussion explores the dynamics of this dilemma by examining the experience of the America's Choice School Design between the years 1998 and 2003. The discussion concludes by arguing that while interveners cannot ignore schools' environments, their efforts to gain leverage on them will never be more than partially effective.

An Examination of the Precision and Technical Accuracy of the First Wave of Group-Randomized Trials Funded by the Institute of Education Sciences

Spybrook, J., Raudenbush, S. W. — Thu, 13 Aug 2009 16:17:00 PDT

This article examines the power analyses for the first wave of group-randomized trials funded by the Institute of Education Sciences. Specifically, it assesses the precision and technical accuracy of the studies. The authors identified the appropriate experimental design and estimated the minimum detectable standardized effect size (MDES) for each study under plausible assumptions about intra-class correlations, covariate-outcome correlations, and explanatory effects of blocking. The MDESs ranged from .18 to .40 for studies funded by the National Center for Education Evaluation and Regional Assistance (NCEE) and from .20 to 1.0 for studies funded by the National Center for Education Research (NCER). They evaluated the technical accuracy by comparing the estimated MDES to the MDES stated in the proposal. Studies funded by the NCEE tended to be very accurate: estimated MDESs and stated MDESs were similar, whereas studies funded by the NCER were less accurate. However, the technical accuracy of the NCER studies improved over time.

Boosting Student Achievement: The Effect of Comprehensive School Reform on Student Achievement

Gross, B., Booker, T. K., Goldhaber, D. — Wed, 06 May 2009 10:03:14 PDT

Between the late 1980s and early 2000s, schools, districts, states, and the federal government devoted enormous resources to the implementation of Comprehensive School Reform (CSR) models. With more than 1.6 billion federal dollars distributed through the Comprehensive School Reform Demonstration (CSRD) project and its successor, the CSR project, states and districts made CSR adoption a central reform strategy for their lowest performing schools. Today, however, federal funding for CSR has dried up, and this policy has been left behind with few explicit efforts to assess the effect of these CSR funds on schools. In this article, the authors look back on this federal reform initiative and the effect it had on Texas students. Using promising analytic techniques for nonexperimental studies to investigate the effects of federal CSR awards on student achievement, the authors find that CSRD funding did not significantly effect students’ reading performance and that its effect on math performance varied across different student types.

What Works in College Outreach: Assessing Targeted and Schoolwide Interventions for Disadvantaged Students

Domina, T. — Wed, 06 May 2009 10:03:14 PDT

By offering information, counseling, and tutoring, college outreach programs attempt to smooth the path between high school and higher education for at-risk students. But do these program work? This paper uses longitudinal data from the Education Longitudinal Study to construct two quasi-experiments to assess the effectiveness of college outreach. The first compares outreach program participants with a propensity score matched sample of program non-participants to measure the effects of targeted college outreach programs. The second assesses the effects of school-wide college outreach programs by comparing students in school-wide outreach high schools with students in a matched sample of high schools that offer no formal outreach. The results suggest that targeted outreach programs do little to change the educational experiences of participating students. However, there is limited evidence to suggest that school-wide outreach programs may have modest "spill-over" effects, improving the educational outcomes of relatively unengaged students at participating schools.

How Does Test Exemption Affect Schools' and Students' Academic Performance?

Jennings, J. L., Beveridge, A. A. — Wed, 06 May 2009 10:03:14 PDT

Analyzing data from a large urban district in Texas, this study examines how high-stakes test exemptions alter officially reported scores and asks whether test exemption has implications for the academic achievement of special education students. Test exemption inflated overall passing rates but especially affected the passing rates of African American and Hispanic students because these students were more likely to be exempted. Furthermore, our results suggest that tested special education students in Grades 3 through 8 performed better academically than they would have if they were not tested. However, taking the high-stakes test provided no academic benefit to special education students in Grades 9 through 11.

Toward Policy-Relevant Benchmarks for Interpreting Effect Sizes: Combining Effects With Costs

Harris, D. N. — Wed, 11 Mar 2009 14:18:15 PDT

The common reporting of effect sizes has been an important advance in education research in recent years. However, the benchmarks used to interpret the size of these effects—as small, medium, and large—do little to inform educational administration and policy making because they do not account for program costs. The author proposes an approach to establishing cost-effectiveness benchmarks rooted in an explicit economics-based decision-making framework and assumptions about the decision-making context. To be considered large, the ratio of effects to costs must be at least as large as the ratios for substitute interventions. Evidence related to class size, prekindergarten, and other interventions is discussed to illustrate the calculation of the cost-effectiveness ratios, how the evidence can be used to develop benchmarks, and how the benchmarks can be useful for researchers and policy makers. The development of benchmarks is intended to encourage cost-effectiveness analysis as a standard part of policy analysis, thereby providing more evidence to increase the validity of the benchmarks and, ultimately, improving policy decisions. Recent cost-effectiveness research in health care policy illustrates the potential value of cost-effectiveness benchmarks in education.

Do Community Colleges Provide a Viable Pathway to a Baccalaureate Degree?

Long, B. T., Kurlaender, M. — Wed, 11 Mar 2009 14:18:15 PDT

Community colleges have become an important entryway for students intending to complete baccalaureate degrees. However, many question the viability of the transfer function and wonder whether students suffer a penalty for starting at 2-year institutions. The authors examined how the outcomes of community college entrants compared with those of similar students who initially entered 4-year institutions within the Ohio public higher education system. Using a detailed data set, the authors tracked outcomes for 9 years and used multiple strategies to deal with selection issues: propensity score matching and instrumental variables. The results suggest that straightforward estimates are significantly biased, but even after accounting for selection, students who initially began at community colleges were 14.5% less likely to complete bachelor’s degrees within 9 years.

Reading Instruction Time and Homogeneous Grouping in Kindergarten: An Application of Marginal Mean Weighting Through Stratification

Hong, G., Hong, Y. — Wed, 11 Mar 2009 14:18:15 PDT

A kindergartner’s opportunities to develop reading and language arts skills are constrained by the amount of time allocated to reading instruction. In the meantime, the student’s engagement in learning tasks may increase if the instruction has been adapted to his or her prior ability through homogeneous grouping. This study investigates whether the grouping effects on kindergartners’ reading growth depend on the amount of reading instruction time and the intensity of grouping. To answer the study’s research questions requires causal inferences about concurrent multivalued instructional treatments. The authors develop a procedure of applying the method of marginal mean weighting through stratification to multilevel educational data. Results from the Early Childhood Longitudinal Study Kindergarten cohort data set lend support to the theoretical hypothesis that when teachers allocate a substantial amount of time to reading instruction, homogeneous grouping helps kindergartners to gain more in reading. The authors find no effect of homogeneous grouping when the total amount of reading time is limited. They also find that the benefit of increasing reading instruction time becomes evident only if kindergarten teachers adapt instruction through homogeneous grouping.

A Randomized Field Trial of the Fast ForWord Language Computer-Based Training Program

Borman, G. D., Benson, J. G., Overman, L. — Wed, 11 Mar 2009 14:18:15 PDT

This article describes an independent assessment of the Fast ForWord Language computer-based training program developed by Scientific Learning Corporation. Previous laboratory research involving children with language-based learning impairments showed strong effects on their abilities to recognize brief and fast sequences of nonspeech and speech stimuli, but generalization of these effects beyond clinical settings and student populations and to broader literacy measures remains unclear. Implementing a randomized field trial in eight urban schools, we generated impact estimates from separate intent-to-treat and treatment-on-the-treated analyses of the literacy outcomes of second- and seventh-grade students who were more generally at risk for poor reading and language outcomes. There were some problems of implementation in the field setting, and the Fast ForWord Language program did not, in general, help students in these eight schools improve their language and reading comprehension test scores.