Analysis of the Oct-Nov 2019 WCPSS MVP Math Curriculum Online Survey Responses

Background

Back in August, 2019, WCPSS staff and the voter-elected School Board gave parents who complained about MVP the collective middle finger when they denied parents' complaint (and subsequent appeal) about the program.  One concession WCPSS made was that they would throw another $125k of taxpayer money at the problem by commissioning an "independent" survey to look at the program and/or its implementation.  They later hired Florida-based MGT Consulting for this task.

MGT conducted their review in the October - November timeframe and that included a 9-question online parent survey which ran October 29 - November 13.  The first few hours of the survey being online was met with outcries from parents, as there were no spaces allowing write-in comments.  MGT quickly fixed that, citing what I remember to be a technical issue.

MGT's Report

The results of the review were presented to the Student Achievement Committee (and hence, the public) on December 16, 2019.  My synopsis of the report was this: MVP is great and wonderful!  In fact,  it's so great and wonderful, we only need to perform SEVEN major corrective actions to make it work! 

The presentation by MGT was quite a sales pitch, and presented all aspects of MVP in only the most positive of light.  It was as though they were given marching orders such as: We're going to keep MVP.  Please make sure your report doesn't contradict that.

Anyway, all positives were emphasized, and any negatives were glossed over and dismissed as though they were known issues already being diligently worked by the WCPSS curriculum staff.

One such area glossed over was the parent survey feedback.  MGT reported that 3125 surveys were submitted.  They boiled the survey responses down to one summary chart, which only showed the most chosen response for each of the 9 questions!  I thought the detailed report may have done the analysis more justice, but it did not.  In my opinion, this is very amateurish at best, and statistical malpractice at worst!

Public Records Request

The next day, I submitted a public records request to WCPSS.  I wanted to see the full survey responses because frankly, I believed that the feedback from as many as 3125 parents deserved more attention and consideration than it was given.

I'll spare you the details, but would you believe I had to fight quite hard to get those results, including emails and School Board speeches?  I felt WCPSS was trying to use a 3rd party to hide public records paid for (and written by) the taxpayers.  Eventually, WCPSS and MGT relented, and on April 17, 2020, I received a 207 page PDF file.  Original version of survey responses (received 4/17/20, 207 pages)

The first thing I did was search for my own input, as I had saved of copy of my own survey answers.  My input was NOT there.  I eventually determined that the survey responses sent to me was a snapshot taken at November 2, 2019 11:23 AM EDT, as noted in the file itself.  This was 11 days before the survey ended, and seemed to only reflect the input of about ~2425 of the 3125 total survey responses reported by MGT.  Seven hundred parents, including me, were not represented in this file.

I reported this to Tim Simmons at WCPSS, and he promised to follow-up.  On June 5, 2020, I received the updated file, which I am calling version 2.  Version 2 of survey responses (received 6/5/20, 292 pages)

This file is better in some ways.  It is better labeled.  It also has fewer odd characters missing, which seemed to confuse the original version.  It also addressed what appeared to be an error related to the comments for question 5, which were omitted from the original version.  One shortcoming of version 2 is that it omitted the write-in comments for question 1, which totaled about 18 pages in the original version.  I assume that is a careless oversight.  Whatever.  Any single page in either of these documents is as damning of MVP as any other.   What's another 18 pages when WCPSS has not heeded the massive volume of MVP rebuke (data, reports, surveys, walk-outs, evidence, public comment, protests) already provided them?

Update 6/22/20: WCPSS Tim Simmons provided me an addendum with the missing comments for question 1 that were omitted from the Version 2 file.  See MGT Parent Survey V2 - Q1 Comments for the additional 20 pages.

Table of Contents for Final Version

Question	Original Version   Received 4/17/20	Version 2 Received 6/5/20
How did you first learn about the MVP Math Curriculum?	Labelled as Q2 Pages 1 - 31	Labelled as Q1 Pages 1 - 9
Answers to “Other (please specify)	Labelled as Q2_4_TEXT Pages 1 - 12	Pages 1 - 9
Write-in comments	Labelled as Q14 Pages 13 - 31	OMITTED but provided in Addendum
What is the difference between the MVP math curriculum and the previous math curriculum?	Labelled as Q3 Pages 32 - 65	Labelled as Q2 Pages 10 - 62
Which statement best describes the level of rigor / challenge provided by the MVP math curriculum?	Labelled as Q5 Pages 66 - 100	Labelled as Q3 Pages 63 - 112
Which statement best describes your experience with the WCPSS mathematics resources website?	Labelled as Q7 Pages 101 - 124	Labelled as Q4 Pages 112 - 140
Which statement best represents your child's experience with the MVP math curriculum?	Labelled as Q8 Pages 125 - 126 Write-in comments OMITTED An error code is shown:  “WIDGET_ERROR.MISCONFIGURED”	Labelled as Q5 Pages 140 - 185
Which statement best describes your child's performance on math assessments since the implementation of MVP Math Curriculum?	Labelled as Q9 Pages 127 - 145	Labelled as Q6 Pages 185 - 211
I have received updates, resources or other communication from my child's teacher about the MVP math curriculum.	Labelled as Q10 Pages 146 - 159	Labelled as Q7 Pages 212 - 230
Since the MVP math curriculum was implemented in my child's school, I have had to spend more time helping them at home.	Labelled as Q11 Pages 160 - 185	Labelled as Q8 Pages 231 - 261
Since the MVP math curriculum was implemented in my child's school, my child has spent more time working on math assignments/homework.	Labelled as Q12 Pages 186 - 207	Labelled as Q9 Pages 262 - 292

After I read a good bit of the document, I saw a trend.  There were many superlatives which seemed to characterize parents' ire towards MVP.  I decided to make my own graph showing the frequency of such words and word forms.

I believe this bar chart to be much more informative about the parent surveys than the chart MGT produced.

Summary of Select Superlatives

Note 6/22/20: This chart is based on the Version 2 file and does NOT include the words found in the addendum provided on 6/22/20.

I hope this report was informative.  Please be sure to vote this November.  All School Board seats are on the ballot and most of them need to be replaced.

MVP Ground-Zero Math Performance Data Exposed, and it Ain't Pretty: An Analysis of American Fork Junior & Senior High Math Trends

An Illustration to Lighten the Mood

Note: Skip ahead to "Background" if you want to jump to the data

What if I told you I had a car to sell you - NO, GIVE YOU - that will bring you an increased feeling of #equity to all your driving needs!?  In fact, you will gain "deeper driving" experiences which enable 21st century transportation skills like you've never seen before!  No more "rote driving!"  We're talking "real world" "critical driving" improvements.   In fact, this new car satisfies all 8 practices of sound driving as identified by the National Council of Car Drivers - get this - by focusing only on ONE of them: Practice #8 - Use the steering wheel.  You heard me right.  By focusing on using the steering wheel the way that I'm going to teach you, you are going to be driving like never before!  I'm willing to GIVE you this car - TODAY - if you are ready to engage in this really popular type of driving discourse.  Soon enough, you'll be driving on DIS-course AND DAT-course!  Get it?  DIS-course?  There's only one catch.  You have to pay ME to teach YOU how to use that steering wheel MY WAY.  And that is the ONLY way you can use it.  Otherwise, Hey, fuggedabout it.    

You say, "All those benefits sound great, but will it help my gas mileage?  I'm getting 58 MPG now, which is pretty good.  And that is my main priority when it comes to transportation!"  

Is that all you think about?  Measuring results?  You're using traditional driving thinking.  I assure you, speaking as an expert in this new drivagogy, you will be delighted in this new "discovery" method of driving the new way.  (Drivagogy is "the method and practice of driving, especially as an academic subject or theoretical concept.")

Let me tell you, since I have been using my new drivagogy method, my MPG improved from 58 to 57.6 in less than 5 years! 

"But wait.  That's not an improvement.  That is basically flat - actually a slight decline," you say.

You got to understand.  The drive performance improvements don't come right away.  You have to be patient.  After a little productive struggle, you'll get the hang of it.   Here - let me offer you this tasty beverage.  Oh Yeaaaaah.  Here at M.V.P. Motors, you have to taste this to believe it.

So, what would you do?  I'm guessing that most sane people would drive, walk or RUN away from such a proposition.  Unfortunately, that little story is closer to the truth than fiction.

Background

Mathematics Vision Project, LLC, was founded in 2013 by principal Travis Lemon.  In addition to being the thought leader behind MVP, Mr. Lemon currently teaches at American Fork Junior High in American Fork, Utah, where he has taught since 2000.   According to Karen Feld, who is also a math teacher at AFJHS, "Travis is an amazing mathematics educator who is focused on the learning and success of all students."  Additionally. Ms. Feld notes, "My team and I use the MVP materials. These materials are excellent. The MVP materials allow students the opportunity to engage in the 8 Mathematical Practices and allow me the opportunity to include the high leverage and research based Effective Teaching Practices (NCTM, 2014) in my classroom. We have had great success and our students demonstrate high levels of mathematical understanding."

While I don't question Mr. Lemon or Ms. Feld in their passions for being teachers, I have a responsibility to look at the data in light of what we are seeing in Wake County.  My thinking is this: American Fork Junior High School has CLEAR ADVANTAGES OVER EVERY OTHER SCHOOL OR SYSTEM ADOPTING MVP.

1. Expertise: As the creator, author, and #1 user of MVP, Mr. Lemon is arguably THE WORLDWIDE EXPERT on the subject.  
2. Buy-In: Additionally, not only is he one of the MVP teachers at the school, he apparently has an adoring set of peers who wholeheartedly embrace the MVP curriculum.  
3. On Demand Help: Furthermore, if there were ever any questions or issues to work through, Mr. Lemon could provide on-demand professional development in the Teachers' Lounge or cafeteria with his peers.  

Certainly, for these 3 reasons, this is a PERFECT SCENARIO in which to deploy MVP and observe the mathematics comprehension bliss it will deliver to students!  I think that most logical people would expect his school to produce the BEST RESULTS POSSIBLE compared to any other in the world.

Disclaimer

I have great respect for all teachers who give their all each and every day for their students.  I know that no curriculum is perfect and no measurement system is perfect.  Sometimes we may disagree about how and why things should be done in the field of "educating the children."  Any observations or critiques I make below are not intended to be aimed at teachers or students, but to the decisions that perhaps should be reconsidered in hindsight.

Utah's Data Gateway

The Utah State Board of Education provides a data gateway for citizens to examine school results.  You can run a report for a school and you will see the school's results in context of up to 19 other schools that are similar.  This "similarity comparison" is very insightful because it groups schools using a "general coefficient of similarity," taking into account about 11 variables contextualizing school attributes and student demographics.

Utah measures % Proficient (percent of students who are grade proficient in a subject) and Median Growth Percentile (MGP).  There is a video which explains MGP, but I will summarize.  Student Growth Percentile (SGP) measures each individual student's percentile compared to similar performers in the prior year.  MGP, however, measures how much students are improving from year to year as a group.  An MGP of 55 would mean that for the population considered, the median SGP was 55.  This would mean that the median SGP student improved better than 55% of his/her peers at the state level.  In short, MGP measures year to year improvement.  Utah's objective is that an SGP of 40 or higher is considered to have made sufficient growth.

2014/15 to 2017/18 Data

So let's look at the 2014-15 data for American Fork Junior High School.  Here you will see the Proficiency is 58% meaning 58% of students were grade level proficient in math.  The MGP measuring growth from the prior year was 53.  Meaning that when we take all the AFJHS students and sort their SGPs, the median one was 53.

If we fast forward to 2017-18, let's see those results.  You will note that there is not much change in proficiency from 2014-15 to 2017-18.  However, the MGP declined from 53 to 48.  Still OK, but declining.

Pulling in the two years in between, the 4 year trend in proficiency is: 58 - 60 - 56.4 - 57.6
The 4 year trend in MGP is: 53 - 52 - 48 - 48.

Pre 2014 Data

Now, when I saw this, I had some concerns.  I am estimating that AFJHS started using MVP around 2013.  After all, I would have expected Mr. Lemon to have a product nearly in hand before forming the Mathematics Vision Project, LLC.  Anyway, the other thing that happened in the 2012-2013 timeframe is "common core".  The data above doesn't go back before 2014.  However, there is another source which shows some older data.  This particular view seems to only be current through 2015-16.

As seen in the graph below, before Common Core, proficiency was on the rise from ~77% or so in 2010 up to around ~90% in 2013.  Then Common Core and/or MVP became the law of the land at AFJHS.  Most data I've seen for other schools shows a decline in scores once Common Core came into play.  So I will not focus so much on that because this report is not about Common Core.  It's about MVP.  And for all intents and purposes, it seems that the 4 years of data available at the original data gateway aligns to MVP years 2-5 at American Fork Junior High School.

Comparing & Contrast with Two Other Schools

To complete this analysis, I looked at two other schools.  The first was American Fork High School.  Approximately 100% of the students from AFJHS feed into the high school of the same name.  Additionally, about 50% of students from another junior high feed into the same AFHS.  This is illustrated as shown.   My thinking was that the MVP students leaving the junior high could also be measured in high school.  Even though we don't know the curriculum of the "other junior high school" about 2/3 of the math students at American Fork High School are from AFJHS.  So the data would be interesting if nothing else.

For the record, I reached out to teachers at the American Fork High School and asked if they used MVP.  They were very nice and responded promptly.  The answers included these comments:

• We do not use MVP. If we do, we use it sporadically, maybe once or twice each term
• We use our own curriculum and some teachers choose to insert an MVP lesson or two into our curriculum where they see necessary but it is not common
• From my brief interactions with it, we as a department saw many issues with it and haven’t ever pushed for it at our school.
• I personally use MVP very little.  At the high school level, I find tasks are a great tool for discovering some concepts, supplementing others, and some just require old fashioned direct instruction.  
• As a department at our high school, we are working to incorporate more tasks because most of us teachers use primarily direct instruction, and we feel that tasks could deepen understanding.

The second other school I looked at was Fairfield Junior High, which is in Kaysville, Utah, in the Davis County District.  Why Fairfield?  If you will refer back to the 2017-18 comparison graphic for AFJHS, you will see Fairfield at the top of the list.  Again, Fairfield is a school which is similar to AFJHS.  In fact, the website shows them as 91% similar.  I wondered how they fared over the prior 4 years, so I pulled that data too.

Let's see the results.  Let it soak in.

Observations

So here are some observations:

• As noted above, you see that American Fork Junior High is relatively flat on Proficiency, and down on Growth (MGP).  
• But look at the high school (where 2/3 of its students are from AFJHS).  Down...  Down... Down... on both Proficiency and Growth.  Just to be clear, we don't have a perfect correlation that says that MVP students at AFJHS are driving the declines at AFHS, but there seems to be some correlation based on the 2/3 of the high school students represented being from the MVP junior high.  If I was a parent of those AFJHS or AFHS students, I would certainly want to know. 
• At the high school, Proficiency has dropped 17.4 points from 53.8 to 36.4.  That is about a 1/3 drop and reflects devastating news to the students and parents affected.
• One might be tempted to think that if the high school would only adopt MVP, then the MVP kids coming from the junior high school would do OK.  But think about that logic.  The direct teaching style is what students will experience in college.  If they are not making the transition from junior high (discovery) to senior high (direct), then I shudder to think how they will handle math in college.
• Growth at this high school is down 16 points from 55 to 39, another drop of nearly 1/3.  The 39 is BELOW what Utah considers acceptable growth.  Is anyone there looking at these results and wondering, "W. T. H?!!"
• Now, we look at Fairfield.  Their proficiency has steadily increased over this same period from 57.5 to 63.7 which is 6.2 points, or a 10.8% increase.  Growth has dropped a tad from 65 to 60, but still the Growth results are outstanding and well over 50.

I'm not shy about reaching out to strangers, so I did.  I contacted teachers at Fairfield Junior High and wanted to find out what their secret to success was!  After all, isn't this one of the ways education leaders should select new ideas and curricula?  Find out what works!

I met with Lori Kalt there, and here's what she told me:

1. A/B block schedule, so we see our students every-other day for 80-minute class periods.
2. We defied the recommendation and push from our district to use their adopted curriculum (Carnegie) and we developed our own curriculum.  (By the way, if you compare Fairfield among all 26 junior high schools in their county - they are #2 in math proficiency for 2017-18.  Lori noted that the #1 school, Kaysville Jr. High, does not use the block schedule for math - so their math class-time is considerably longer.)
3. We employ lots of best practices in our curriculum which includes lots of direct instruction, skill-based practice for fluency (on skill-based concepts), whole-class activities and dialogue to help students grasp and understand conceptual things, analyzing mistakes, fine-tuning understandings using mini white boards, and daily short quizzes on each standard. 
4. We use standards based grading and students have multiple opportunities to show mastery on concepts - not unlimited opportunities but students can retake quizzes a few times
5. We make sure all students are fluent on foundation skills such as solving equations, constructing equations, graphing equations etc.

She also noted, 

We are currently receiving a LOT of pressure from our district to employ the use of tasks more frequently.   I work with an amazing group of math teachers and we are all like-minded and collaborate a LOT with each other.  We are hesitant to wholly employ the latest fad(s) in education which right now seems to be tasks, discovery, project-based learning, and personalized/online learning.

Conclusion

So there you have it.  Who would have thought that having human teachers create a curriculum from scratch could beat a bill of goods and services which has cost Wake County over $1.5M?

Who would have thought that teaching using a mixture of various crazy ideas such as:

• direct instruction, 
• skill-based practice for fluency (on skill-based concepts), 
• whole-class activities and dialogue to help students grasp and understand conceptual things, 
• analyzing mistakes, 
• fine-tuning understandings using mini white boards, and 
• daily short quizzes on each standard 

would have resulted in not only UP-TRENDING results, but clear leadership compared to similar schools?

Who would have thought that those results would be far superior to ground-zero of the latest and greatest trend in so-called innovative (yet-unproven) constructivist mathematics pedagogy?  

I tell you who: ME... and many other parents and teachers in Wake County and around the US and Canada who know better than to risk our students' futures on a tweet filled with feel-good hashtags, or on a chance meeting at a dog and pony show at a math conference.

You know what WCPSS should do NOW that they know the truth about MVP?

Fuggedabout it.  

Another One Bites the Dust: MVP Success in Modesto, CA Debunked - "Real World Problem Solving" Sub-score Hardest Hit

"MVP will teach students to solve real world problems!  MVP is equity-based!" they monotonously repeat and tweet and repeat and tweet.  So many times that some actually believe it.

Summary

In this 2017 video, at the 1min 30sec mark, you will hear Ginger Johnson, Associate Superintendent Educational Services, talk about the gains experienced by the new integrated math program in the Modesto District of Stanislaus County, California, where MVP was adopted beginning in 2014 (3 years ahead of WCPSS):

"Educational Services is pleased to report we have seen steady gains in growing student success in Math.  This is evident in the increasing number of students taking advanced 4th year courses as well as students' performance on SBAC math assessment....  we have seen a small steady increase in math proficiency."

Oh, have you?

Background

One uniqueness of this school district is that over 50% of its students who take this assessment are Hispanic / Latino.  So if the MVP claims are true, we should see clear improvements in overall performance and this large subgroup.

California public school test assessment results are available from the CAASPP website.  They administer a state level math assessment in 11th grade.  Scores fall into 4 buckets: Standard Exceeded, Standard Met, Standard Nearly Met, and Standard Not Met.

In the charts below, I am showing results for California, Stanislaus County, and Modesto District, and then comparisons between the district vs. state and district vs. county. As the numbers show, there are gradual downturns in both the raw numbers and the comparison numbers.

District vs County & State - ALL

I like to compare data to a control group, such as a larger population in similar circumstances.  In California, my understanding is that about 2/3 of the state has adopted integrated math.  This is not necessarily MVP, but may explain the declines in state scores.

Sub-scores Not Good

So this is where it gets interesting.  The California test also captures, 3 math sub-scores which are evaluated on a below, near, above standard basis:

1. CONCEPTS & PROCEDURES: Applying mathematical concepts and procedures
2. PROBLEM SOLVING & MODELING/DATA ANALYSIS: Using appropriate tools and strategies to solve real world and mathematical problems
3. COMMUNICATING REASONING: Demonstrating ability to support mathematical conclusions

Now, if you have paid attention to MVP promotional materials, you will know that the big thing they brag about is "critical thinking" and "deeper learning" and the ability to "solve real world problems".   So, to me, that sounds like #2.

Let's look at the data:

Here's #1, CONCEPTS & PROCEDURES: Applying mathematical concepts and procedures

Here's #2, PROBLEM SOLVING & MODELING/DATA ANALYSIS: Using appropriate tools and strategies to solve real world and mathematical problems.

Here's #3, COMMUNICATING REASONING: Demonstrating ability to support mathematical conclusions.

So the silver lining in this - if you're an MVP zealot - is that in sub-score 3 "Communicating Reasoning", there was only a 1% increase in the number of students BELOW STANDARD!  Maybe all that group collaboration is paying off.  Hooray!!   

Meanwhile, the bread & butter of MVP - "deeper learning" and solving "real world problems" - saw an increase in BELOW STANDARD from 39% to nearly 51%!  (That's 1 more kid out of 10 who can't solve real world problems.  Darn!).  

Lagging behind, "Concepts and Procedures" continued its BELOW STANDARD climb from 56% to 62%.   

Hispanic / Latino Subgroup Has Similar Woes

District vs County & State - Hispanic / Latino Subgroup

Sub-scores for Hispanic / Latino Subgroup

So that's it for now.  Similar results as for the ALL data.

By the way, you can jump ahead to the 37min mark to hear from several teachers who speak about MVP.  There are a few fans, and that is expected because some get into it.  But the majority are against.  It's not for everyone as I explained in Myers-Briggs and Why MVP Doesn't Work at Scale.

MVP Eradicates WCPSS's Performance Lead over NC in "Economically Disadvantaged" and "Limited English Proficiency" Subgroups

"MVP is low entry high ceiling," they tell us.  But how are subgroups of Economically Disadvantaged Students (EDS) and Limited English Proficiency (LEP) faring in Wake County?  Let's take a look.

Source

All data quoted here is available to the public at the Department of Public Instruction's (DPI's) testing results webpage.  We encourage you to check the results yourself.

Background

In Wake County Public School System (NC), Math 1 courses adopted MVP last school year 2017-18.  NC captures 2 metrics for the Math 1 EOC, Grade Level Proficiency (GLP) and College and Career Readiness (CCR).  There was a change to the way the EOC was reported in 17-18, making comparisons to prior years difficult without math acrobatics that are the exclusive intellectual property of WCPSS.  Therefore, I will call the raw data for 15-16 and 16-17 "Apples" and the raw data for 17-18 "Orange".

However, there is meaning in comparing Wake to the state of NC, and then trending that number.  Wake County has been one of the top performing school systems in the state for many years, consistently outperforming the state in all subgroups.  But, as you will see here, that gap is closing for the "all students" data set, and even more so for the EDS and LEP groups, especially in 17-18, after one year of using supposed "free" OER resource MVP.

"What's the Cliff Notes Version?" you ask.  

The pictures speak for themselves.

For Group 1, Economically Disadvantaged, Wake once led NC by 1.9% and 4.5% for GLP and CCR, respectively.  Now, we trail the state by 9.3% and 11.8%!  Thanks MVP!

For Group 2, Limited English Proficiency, Wake once led NC by 27.8% and 43.2% for GLP and CCR, respectively.  Now, the gap has crashed and we are about dead even at 0.4% and 5.0%!  Thanks MVP!

For Group 3, All Students, Wake once led NC by 19.2% and 26.9% for GLP and CCR, respectively.  Now, that gap has continued to decline to 13.4% and 18.9%!  Thanks MVP!

Conclusion

I continue to be skeptical of all WCPSS's Year 1 (2017-18) MVP data claims due to the secret mystery algorithm used to calculate the performance numbers, which were reported without the context of the state numbers.  But mainly, due to the widely irregular adoption we have seen of MVP in Wake County, how can anyone make cause-and-effect claims, including me?  That said, it's the only data we have, and this report shows that EDS and LEP students are getting the shaft.

Author: Blain Dillard

Why My MVP Golf Score Improved, and other Exaggerations

If I told you the average golf score for Wake County golfers improved 1.9 strokes last year as compared to the prior year, and then told you that was due to the new Golf Galaxy store on Hwy 55 in Cary which opened in 2016, what would you think of my statistical analysis abilities?  My guess is that you might question my logic.  You might wonder if all golfers in Wake County visited that Golf Galaxy, and if so, did they purchase any products or services (such as lessons) that would have helped their scores?  How much golf practice did they do in their own time?  Did they receive any other golf lessons or tips from others?  Did they watch any golf training videos on YouTube?  You might wonder how the golf scores fared for golfers who visited other golf stores, if at all.  What were the trends in prior years?  Up?  Down?  You might also wonder how the average scores in NC were overall for the same period.  You would certainly want to look into all of those matters before making such a cause -> effect claim about such an improvement.  To do so otherwise would be an irresponsible use of golf statistics!

Too bad WCPSS and Mathematics Vision Project doesn't approach stats like you and I would.  They published this graphic on Twitter in November 2017 boasting of WCPSS's improvement in Math 1 proficiency due to MVP math, a "free OER" according to the statement.

Side note: "OER" mean "open educational resource".  "Free" usually means "of no cost" but that is a different story. 

For this story, I'd like to say it's easy to verify the data using the NC Department of Public Instruction data website.  But that's not 100% true, because in 2017-18 school year, the DPI changed the way it calculates the EOC for Math 1.  Therefore, comparisons to prior years are not as easy.

Cases in point, when I looked up the data, the grade level proficient (GLP) and college & career readiness (CCR) numbers seem to be way down from 2016-17 to 2017-18 - for both WCPSS and State of NC.  This is because the 2017-18 data is only reporting high school results, whereas in prior years, the middle school Math 1 EOCs were wrapped into the metrics.  Those middle school Math 1 EOCs are typically higher (because those are the kids taking higher math sooner), so omitting them, brings the average down.

I have asked WCPSS for an explanation as to how they arrived at the 1.5 and 1.9 numbers, but have yet to receive a response other than a March 3, 2019 acknowledgement of the change confirming "an additional level of analysis was required at the district level in order to provide a more accurate cross-year comparison." Additionally, "Staff is working to get more information on the two data points in question and will follow up with you shortly to provide additional details and/or data that may be helpful."  Well, THAT is what I want to see!

Anyway, we can use these numbers as they are and still make a point or two.  Let's take a look at the 2 graphs, one for CCR and the other for GLP.   For sake of simplicity, let's call the data points for 14-15, 15-16, and 16-17, the apples.  And the 17-18 data points, the oranges.

Both graphs show the 3 years preceding 17-18, the first year MVP was implemented.  During the apple years, the number went up for both WCPSS and the state.  Both graphs also show the apples to oranges drop from 16-17 to 17-18 due to the change in how the calculation was done.  So there's no benefit in comparing apples to oranges.

But there is benefit in demonstrating that even if the 1.9% and 1.5% were completely true and verifiable, those gains were lower than the PRIOR year, when both metrics rose 2.3%!  

This goes back to the golf score average improving because of the new Golf Galaxy.  If my score was already getting better the prior year, how can I say the new Golf Galaxy helped me this year?  I can't!

Additionally, let's look at the trends comparing WCPSS to the state overall.  For CCR, the gap between WCPSS and the state has declined from a peak of 13.4 in 15-16 to a recent 8.8 in 17-18.  For GLP, the gap has closed from 11.9 to 7.7.

So just to be clear.  I am comparing apples to apples for the 3 years shown 14-15, 15-16, and 16-17.  And I am comparing oranges to oranges in 17-18.  The gap is closing between WCPSS and the state.  Is that because the state is doing better?  Or something else?

So in the end, I don't think it matters how WCPSS arrived at the 1.5 and 1.9 numbers.  Either way, the gains are no better than the prior year, and the gap with the state is closing.  In other words, I bet that if we applied the "additional level of analysis" at the state level, we would find their gains to be more than 1.5 and 1.9 for 17-18, further trivializing the boastful graphic posted by WCPSS and MVP.

Therefore, I declare this graphic from WCPSS claiming an improvement in proficiency in year one DEBUNKED!  Now, go hit 'em long and straight - and NO three-putting!

UPDATE 3/29/19:
Dr. Denise Tillery provided me with data I requested showing this data in context and an explanation why the DPI data is not usable as-is and has to be tweaked in order to make sense of it.  I'll save that debate for another day because it's irrelevant here.  Here is the data for the 2 years leading up to year 17-18, which was the first year of MVP.

So this is eye opening because, as I expected, the 1.5% and 1.9% claims were nothing remarkable.  In fact, in the prior year, the gains were 4.6% and 3.2%.  So the "rate of change" declined.  The 1.5 and 1.9 were nothing to brag about.  Plus, take into account that MVP adoption adherence is wildly irregular across the county AND the fact that parents are hiring tutors when their kid suddenly has become a math un-wiz and can't do problem #1 on homework.  So there are always extenuating factors which contribute to data.

When I confronted WCPSS leaders about this in a face to face meeting on April 8, 2019, they admitted the misleading nature of the graphic from MVP, but claimed their own chart was intending to say "The numbers were actually up, when usually we have a decline when introducing a new curriculum"  So they were just pleased the numbers didn't go down!  What a low bar to set in an experiment conducted on our children.  So let's think this through...Our performance is increasing - let's introduce a new curriculum which we know will bring it down!

In a separate communication, WCPSS admitted the cost of "free OER resource" MVP to be $1.4M.

So, here is the updated graphic.   At that meeting, WCPSS said they would ask/demand MVP take down the twitter post which misrepresents the WCPSS chart.  It's still there as of 4/11/19.

UPDATE 4/24/19:  The tweet has been removed from the MVP website!!!  Thank you WCPSS for doing what's right!  Now if only Chapel Hill - Carrboro would do the same.

Author: Blain Dillard

Guest Author: Berkeley Parent Advocate Shares MVP Experiences

Wake parents are partnering with Berkeley parents as well as those from Anacortes, Washington, to learn about their experiences in advocating for math education for their students impacted by MVP.  In this article, Berkeley High School (California) parent advocate and Wake MVP Parent guest author Donna Storey tells the history of BHS's MVP journey.  

Cliff Notes Version

1. Berkeley High School implemented MVP beginning in 2015 hoping to close its long standing achievement gap.   
2. Students frustrated and confused, losing confidence in math.  School system blames children. 
3. School system attempts to hide data, but is eventually required to release data by a California Public Records Act request.
4. D/F grade rates increase, even bolstered by 54% of surveyed students receiving private tutoring.
5. Parents created support websites and tutoring programs.

Main Article

In the fall of 2015, the Berkeley High School math department launched a brand-new Common-Core-aligned curriculum for the class of 2019: Mathematics Vision Project  Many hoped this new approach to teaching math would help close the long-standing achievement gap at the school.

I was serving as the secretary of the Academic Choice Advisory Council, a sort of PTA for Berkeley High’s largest “small learning community.” As early as the fall of 2015, parents were coming to our monthly meetings to express concern that their children were struggling with the new curriculum. With no textbook and little guidance from the teachers, students found homework confusing and frustrating. Students who had once excelled in and loved math lost confidence in their abilities. During information nights, math department representatives shared very little information about the new curriculum, but merely assured parents that everything was on track and this new approach would end up being a great improvement. Many parents and students felt equally confused in the second year of MVP, but most assumed it must be their child’s personal issue.

By the winter of 2018, the class of 2019 was now taking the junior-year MVP class called Math 3. The D/F rate for these students was reportedly at an all-time high. Although my children had both graduated from Berkeley High, I heard stories from parents of juniors that made me very concerned.

I requested the D/F rates from the Berkeley Unified School District by means of the California Public Records Act and received the following information, updated March 7, 2019:

Aggregate D/F Rate for MVP Math Courses at Berkeley High by Semester

Course	2016-17 (S2)	2017-18 (S1)	2017-2018 (S2)	2018-19 (S1)
Math 1	28.4%	25.4%	28%	26.3%
Math 2	25.2%	26.7%	26.9%	23.8%
Math 3	not yet offered	18.1%	20.7%	6%**

A request for more detailed math performance data in the years before MVP was denied by the Berkeley Unified School District, but the “Update on Common Core Mathematics” (item #12.2) presented to the Superintendent of the Berkeley Unified School District on March 22, 2017 contained the following data, which confirms that there has been a notable increase in D/F rates since the introduction of MVP in the 2015-2016 school year:

TABLE 1: D/F Rates for all 9^th grade math students semester 1 (Algebra I or Geometry)

Academic Year           All students

2013-2014                15%

2014-2015                17%

TABLE 2: Math 1 D/F rates over 1.5 years (this does not include Advanced Math 1 or Math 1X) [first year of MVP at Berkeley High School]

Term                           All students

Sem 1 15/16                         16.7%

Sem 2 15/16                         19.4%

Teachers and administrators continued to reassure parents all was well and they were handling any issues behind the scenes. Students of means relied on private tutors (54% of the 132 parents who took a community survey reported they’d hired a private tutor). Other concerned parents came together to create support websites and encourage the math department to make videos to explain concepts to students, although teachers did not emphasize the availability of this material. A group of parents started a free Sunday afternoon tutoring program to supplement after-school tutoring by teachers at Berkeley High. The Berkeley Unified School District hired a math coordinator to oversee math instruction in the Universal Ninth Grade program beginning with the class of 2022 to provide students with more support. We are also requesting that the District provide an update to the 2017 report to give the community vital information on student performance.

MVP is now in its fourth year and many students and parents at Berkeley High are still struggling and feel their challenges are not being taken seriously. When a parent group from Wake County, North Carolina reached out to Berkeley High Parent Advocate, the issues they were experiencing with the launch of MVP were sadly familiar: student frustration and confusion, many losing confidence in math; a lack of data on student performance so that the impact of MVP was hidden from the community; and assurances that MVP was a cutting-edge curriculum and any problem was the fault of students.

I hope that students and families in all school districts who are using MVP can connect, share their stories, and work to open a dialogue with MVP and their school districts. MVP is a new curriculum, and no matter how carefully designed, the reality of student experiences must be taken into account in order to best serve our kids. For example, EdReports found that MVP did not meet expectations for differentiated instruction for diverse learners and only partially meets expectations for gathering information about students’ prior knowledge and providing guidance for remediation. We must not take MVP as a perfect program that cannot be changed or challenged, but rather as an experimental approach that will benefit from student feedback and careful monitoring by teachers and administrators. Indeed, one of the unfortunate side-effects of the lack of transparency and dialogue is that teachers and administrators also suffer unnecessary stress when they take a defensive rather than collaborative position in relation to the community.

I sincerely hope we can work together to support math success for all of our students.

**I will be encouraging the community to ask for more information on the dramatic decrease in D/F rates in Math 3. We do know that the Math 3 staff was changed this year, and two teachers known to be especially demanding were reassigned to other courses in 2018-2019.

Donna Storey

Berkeley High Parent Advocate