Reposted with permission of the American Federation of Teachers
http://www.aft.org
September, 1996
            A Critical Analysis of the Greene-Peterson-Du Paper
___------------------------------------------------------------------------
Since 1990, a voucher experiment in Milwaukee has allowed more than 2,000
low-income students to go to private, nonreligious schools at public
expense. Jay Greene, Paul Peterson, and Jiangto Du (GPD) recently released
a study arguing that Milwaukee private school voucher students do better,
as measured by standardized reading and math test scores, in the third and
fourth years of the program than a group of comparable students in the
Milwaukee public schools. This study dismissed a four-year evaluation by
Professor John Witte, who concluded that the voucher students did not do
better than comparable public school students in any year of the program to
date. Witte's study has been in the public domain and has passed social
science muster for four years. The GPD study has not. Unfamiliar with
statistics, the media have uncritically trumpeted GPD's "conclusion" about
a "private school advantage." We provide the following facts in the
interest of a more informed public discussion of the GPD study and the
school voucher issue. These points do not exhaust the criticisms to which
the study is subject.
1. The GPD study was not subjected to formal peer review before being
publicized.
Before publicizing studies, scientists and social scientists customarily
submit their works to professional journals where they can be subjected to
a formal peer review process. This way, when egregious errors are noted,
authors can revise their work. GPD, in a highly unorthodox move, skipped
the peer review process and released their paper directly to voucher
advocacy groups and to editorial boards of newspapers with an avowed
pro-voucher agenda. Not surprisingly, these groups trumpeted the
"conclusion," and the media, unfamiliar with statistics, uncritically
followed suit. Every researcher who has since examined the study and gone
public has demolished it.
2. The GPD study has serious methodological flaws. GPD's main analysis--the
one they base their conclusions on--violates rules that even a Statistics
101 student would be held accountable for observing. For example:
     + The two student groups being compared are not the same. Yet in
     their main analysis, GPD do not control for such differences as
     parent education level, even though they acknowledge that such
     differences could bias their results.
     GPD claim that the Milwaukee voucher program provided a natural
     experiment by "randomly assigning" students to two groups: those
     selected to participate in the choice program (the experimental
     group) and those not selected (the control group). Because the
     process was "random," the authors argue that these two groups
     share similar characteristics. However, the student selection
     process was not random; the private schools retained the right to
     choose their students. Moreover, GPD's own back-of-report numbers
     show that the two groups were not similar, especially on parent
     education level, one of the most significant out-of-school
     influences on achievement research has ever found. The students
     selected into the private schools had mothers (only mothers'
     education level is presented) who had some college, while the
     students who ended up in public school had high school dropout
     mothers. GPD even acknowledge that you can't look at academic
     achievement gains without controlling for parent educational
     level. Yet in their main analysis, they disregard this fact; they
     do not control for parent educational level or any other
     out-of-school influences, or even ability/prior achievement--all
     of which is standard operating procedure for education
     researchers. Even so, their results are barely statistically
     significant for math and not at all so for reading, despite their
     claim.
     + In a later back-of-the-report analysis, GPD do control for
     parent education level and their so-called "private school
     advantage" disappears.
     As stated above, GPD, in their main analysis, do not control for
     parent education level or any other out-of-school influences.
     However, in a later analysis, they do control for mother's
     education. Their finding is then identical to Witte's: no
     difference in achievement between voucher students and their
     public school peers. How do GPD explain this? In their report,
     they argue that this analysis is meaningless because their sample
     size was so small--but small sample size never stops them from
     hailing the findings that they like.
     + As a result of high attrition rates among both the selected
     private school students and the non-selected public school
     students, the two groups being compared become even more
     dissimilar over the course of the four-year study. Thus, by the
     critical third and fourth years--the years GPD claim the "private
     school advantage" kicked in--the comparison groups, which were
     not even comparable to begin with, are about as similar as apples
     and nails.
     GPD's report leads us to believe that they base their conclusions
     on the 1,356 students who were selected into the choice program
     and the 693 students who were eligible but not selected--a total
     of 2,049 students. However, because of high attrition rates in
     both the selected and non-selected student samples (attrition
     rates averaged 30% a year among selected students in choice
     schools), the number of students GPD look at shrinks dramatically
     over the four years. Thus, by the third year, GPD are looking at
     the math scores of only 310 students and reading scores of only
     309 students. And by the all-important fourth year, these numbers
     are 110 for math and 108 for reading--with no indication of how
     many are the selected students and how many are the non-selected
     students. According to Witte's data (which GPD had), such
     pitifully small numbers mean that in some cases, GPD based their
     conclusion on as few 0-3 students per grade. Basing public policy
     on these results would be like urging a whole population to take
     a new medicine when you've only tried it out on a small
     group--and found that it didn't even work with them!
     This shrinking data pool raises other important questions,
     mainly: Who stayed and who left? And how does the remaining
     sample of selected and non-selected students compare to the
     original sample? Witte's earlier evaluations show that the
     selected students who stayed in choice schools had higher prior
     test scores in their public schools and more involved parents
     than the selected students who dropped out or were pushed out.
     (Note: The private schools have re-admission committees that
     "re-select" students from year to year.) At the same time, the
     non-selected students who stayed in public schools were poorer
     and came from less educated families than the non-selected
     students who left. (Note: The "disappearance" of non-selected
     students was due to their moving out of the district or attending
     a private school outside the auspices of the voucher program.)
     This means that by the third and fourth years, GPD were comparing
     the cream of the crop of students selected into the private
     schools with the bottom of the barrel of non-selected students, a
     discrepancy that clearly favors superior results for the private
     school students. Because GPD do not control for any of these
     student background differences--income, parent education level,
     prior test scores, parent involvement--their analysis is
     worthless for discerning public/private school differences.
     + The results of GPD's main analysis are not even statistically
     significant, using conventional social science standards of
     statistical significance.
     GPD stack the deck by using a statistical standard that is much
     lower than researchers ordinarily use or accept. When researchers
     employ statistics to show that a program works, they look for
     results that are "statistically significant." This means there is
     little likelihood that their findings are a result of error or
     other factors not under investigation in the study. GPD do not
     talk much about "statistical significance" because their
     findings--with one exception--are not statistically significant.
     They talk, instead, about results that are "substantially
     significant" or "substantially important." These are made-up
     terms that have no meaning among researchers. But they do deceive
     the public into thinking that GPD have proven their case.
3. GPD include only a small number of the private schools that participated
in the voucher program.
The GPD study looks at only three schools, but the number of private
schools in the program has ranged from a low of 11 to a high of 16 or 18.
(Even more private schools were eligible to participate, but declined.)
Four of these schools have closed, three under a cloud of financial and
educational irregularities. The head of one voucher school recently under
investigation fled the scene and was arrested in Texas. When you're
promoting a market approach to education, a 25 percent failure rate is an
important piece of information.
4. GPD cannot justify their dramatic claim that if "similar success could
be achieved for all minority students nationwide, it could close the gap
between white and minority test scores by at least a third, possibly more
than half."
Even if GPD's methods were sound and their results statistically
significant (and they are not), this is an extraordinary claim, given that
GPD only look at three participating private schools in their study, and
none of these schools enrolls whites. Two of them are Afro-centric and one
is centered on Hispanic culture. Although there are some whites in the
voucher program, GPD drop these students from their analysis.
5. Since comparisons between private and public school students were first
made in the 1960's, there has not been a single study that supports a
private school advantage once family background (and, at the secondary
level, course-taking) is taken into account.
The GPD study stands in relation to all other studies of this issue in the
same way that an analysis purporting to show that the earth is flat stands
in relation to hundreds of years of scientific evidence showing that the
earth is round.
School vouchers are an important and contested public policy issue.
Children's futures are on the line, and citizens and policymakers depend on
honest evaluations to help them make wise decisions. Given the stakes, it
behooves researchers involved with this issue to do their work with the
utmost care and integrity. The work of GPD ignores this ethical necessity.
Chuck Beams
cbeams@dreamscape.com
http://www.dreamscape.com/cbeams
___
* UniQWK #5290* A lightning bolt partitioned my hard drive.
--- Maximus 2.01wb
---------------