Why My MVP Golf Score Improved, and other Exaggerations

If I told you the average golf score for Wake County golfers improved 1.9 strokes last year as compared to the prior year, and then told you that was due to the new Golf Galaxy store on Hwy 55 in Cary which opened in 2016, what would you think of my statistical analysis abilities?  My guess is that you might question my logic.  You might wonder if all golfers in Wake County visited that Golf Galaxy, and if so, did they purchase any products or services (such as lessons) that would have helped their scores?  How much golf practice did they do in their own time?  Did they receive any other golf lessons or tips from others?  Did they watch any golf training videos on YouTube?  You might wonder how the golf scores fared for golfers who visited other golf stores, if at all.  What were the trends in prior years?  Up?  Down?  You might also wonder how the average scores in NC were overall for the same period.  You would certainly want to look into all of those matters before making such a cause -> effect claim about such an improvement.  To do so otherwise would be an irresponsible use of golf statistics!

Too bad WCPSS and Mathematics Vision Project doesn't approach stats like you and I would.  They published this graphic on Twitter in November 2017 boasting of WCPSS's improvement in Math 1 proficiency due to MVP math, a "free OER" according to the statement.

Side note: "OER" mean "open educational resource".  "Free" usually means "of no cost" but that is a different story. 

For this story, I'd like to say it's easy to verify the data using the NC Department of Public Instruction data website.  But that's not 100% true, because in 2017-18 school year, the DPI changed the way it calculates the EOC for Math 1.  Therefore, comparisons to prior years are not as easy.

Cases in point, when I looked up the data, the grade level proficient (GLP) and college & career readiness (CCR) numbers seem to be way down from 2016-17 to 2017-18 - for both WCPSS and State of NC.  This is because the 2017-18 data is only reporting high school results, whereas in prior years, the middle school Math 1 EOCs were wrapped into the metrics.  Those middle school Math 1 EOCs are typically higher (because those are the kids taking higher math sooner), so omitting them, brings the average down.

I have asked WCPSS for an explanation as to how they arrived at the 1.5 and 1.9 numbers, but have yet to receive a response other than a March 3, 2019 acknowledgement of the change confirming "an additional level of analysis was required at the district level in order to provide a more accurate cross-year comparison." Additionally, "Staff is working to get more information on the two data points in question and will follow up with you shortly to provide additional details and/or data that may be helpful."  Well, THAT is what I want to see!

Anyway, we can use these numbers as they are and still make a point or two.  Let's take a look at the 2 graphs, one for CCR and the other for GLP.   For sake of simplicity, let's call the data points for 14-15, 15-16, and 16-17, the apples.  And the 17-18 data points, the oranges.

Both graphs show the 3 years preceding 17-18, the first year MVP was implemented.  During the apple years, the number went up for both WCPSS and the state.  Both graphs also show the apples to oranges drop from 16-17 to 17-18 due to the change in how the calculation was done.  So there's no benefit in comparing apples to oranges.

But there is benefit in demonstrating that even if the 1.9% and 1.5% were completely true and verifiable, those gains were lower than the PRIOR year, when both metrics rose 2.3%!  

This goes back to the golf score average improving because of the new Golf Galaxy.  If my score was already getting better the prior year, how can I say the new Golf Galaxy helped me this year?  I can't!

Additionally, let's look at the trends comparing WCPSS to the state overall.  For CCR, the gap between WCPSS and the state has declined from a peak of 13.4 in 15-16 to a recent 8.8 in 17-18.  For GLP, the gap has closed from 11.9 to 7.7.

So just to be clear.  I am comparing apples to apples for the 3 years shown 14-15, 15-16, and 16-17.  And I am comparing oranges to oranges in 17-18.  The gap is closing between WCPSS and the state.  Is that because the state is doing better?  Or something else?

So in the end, I don't think it matters how WCPSS arrived at the 1.5 and 1.9 numbers.  Either way, the gains are no better than the prior year, and the gap with the state is closing.  In other words, I bet that if we applied the "additional level of analysis" at the state level, we would find their gains to be more than 1.5 and 1.9 for 17-18, further trivializing the boastful graphic posted by WCPSS and MVP.

Therefore, I declare this graphic from WCPSS claiming an improvement in proficiency in year one DEBUNKED!  Now, go hit 'em long and straight - and NO three-putting!

UPDATE 3/29/19:
Dr. Denise Tillery provided me with data I requested showing this data in context and an explanation why the DPI data is not usable as-is and has to be tweaked in order to make sense of it.  I'll save that debate for another day because it's irrelevant here.  Here is the data for the 2 years leading up to year 17-18, which was the first year of MVP.

So this is eye opening because, as I expected, the 1.5% and 1.9% claims were nothing remarkable.  In fact, in the prior year, the gains were 4.6% and 3.2%.  So the "rate of change" declined.  The 1.5 and 1.9 were nothing to brag about.  Plus, take into account that MVP adoption adherence is wildly irregular across the county AND the fact that parents are hiring tutors when their kid suddenly has become a math un-wiz and can't do problem #1 on homework.  So there are always extenuating factors which contribute to data.

When I confronted WCPSS leaders about this in a face to face meeting on April 8, 2019, they admitted the misleading nature of the graphic from MVP, but claimed their own chart was intending to say "The numbers were actually up, when usually we have a decline when introducing a new curriculum"  So they were just pleased the numbers didn't go down!  What a low bar to set in an experiment conducted on our children.  So let's think this through...Our performance is increasing - let's introduce a new curriculum which we know will bring it down!

In a separate communication, WCPSS admitted the cost of "free OER resource" MVP to be $1.4M.

So, here is the updated graphic.   At that meeting, WCPSS said they would ask/demand MVP take down the twitter post which misrepresents the WCPSS chart.  It's still there as of 4/11/19.

UPDATE 4/24/19:  The tweet has been removed from the MVP website!!!  Thank you WCPSS for doing what's right!  Now if only Chapel Hill - Carrboro would do the same.

Author: Blain Dillard