Look No Further! Even More Analysis of MVP Ground Zero in American Fork, Utah

British prime minister Benjamin Disraeli once said: "There are three kinds of lies: lies, damned lies, and statistics."  I have frequently debunked data showing supposed MVP "successes" by showing how it was used out of context (see Why My MVP Golf Score Improved, and other Exaggerations), or was unsupported by the publicly available data (see MVP Math Claims about Gains in Chapel Hill - Carrboro Debunked!).

A few months ago I wrote about math proficiency data found in Utah in MVP Ground-Zero Math Performance Data Exposed, and it Ain't Pretty: An Analysis of American Fork Junior & Senior High Math Trends.  Recently, the original report I did about these Utah schools was called into question.  The purpose of this article is to elaborate on why I believe this data is golden and should bring grave concern to those hopeful that MVP will ultimately prove to be a success.  As time has passed, I am more and more convinced that THIS is THE most important MVP dataset to examine with care.

RECAP

As noted in the prior article, American Fork Junior High is ground zero for MVP because this is where the MVP founder and one of its authors is a math teacher.  Certainly, there can be no other school in our solar system which has a better MVP situation:

1. Worldwide subject matter & thought leader about MVP as a practitioner of the MVP "craft"
2. The 9th grade math teachers love the program and buy-in to it, according to one teacher there ("the materials are excellent")
3. When in doubt, any teacher certainly can obtain on-demand professional development by merely asking the founder for some pointers.  (Whereas the rest of us poor schmucks have to pay tax dollars to fly the MVP founder and team back to Wake County for more teacher refreshers on "How to Properly MVP")

WHAT'S NEW

I decided to take another look at what's going on in Utah.  In my prior article, I examined math proficiency scores available to the public at Utah's state education website.  I looked at data for 3 schools in particular:  

One was the founder's home school (American Fork Junior High).  The other was the high school fed by that school (American Fork High).  Both schools are in American Fork, Utah, which is north of Provo.  And the third was Fairfield Junior High in Kaysville, UT, which as the other article explains, was #1 in a list of 20 junior high schools similar to American Fork Junior High in 2017-18.  Kaysville is north of Salt Lake City.

The two American Fork schools are in the Alpine District, which has 12 junior high and 9 high schools, and ~79k students.   Fairfield is in the Davis District about an hour away, which has 17 and 9 junior and senior high schools, respectively, and has ~72k students.  

By comparison, Wake County Public School System has about 36 and 28 middle and high schools, respectively, with 160k students (Source: https://www.wcpss.net/Domain/100)  So we are a little bit larger than both of these districts combined, in number of students and middle/junior/senior schools.

It's worth noting that WCPSS is considerably more diverse by race and ethnicity measures, and has a larger percentage economically disadvantaged and English Language Learners student groups than either of the Utah districts examined.

As noted in the prior article, American Fork Junior High School feeds American Fork High School exclusively.  This was told to me by a teacher at the high school.  This makes sense, based on the fact that the high school (2365 students) is only slightly larger than the junior high school (1962).  In fact, the school attendance coverage map (source: https://www.publicschoolreview.com/american-fork-junior-high-school-profile ) confirms that since the American Fork High School map completely contains the American Fork Junior High coverage map.  Portions of students from another junior high school make up the remainder of the American Fork High School population.  It might be worth learning more about the other junior high, but I've learned quite a bit with what is here already.

EATING YOUR OWN DOG FOOD

As a computer scientist and manager with 30+ years experience in the I/T industry, I know how important it is for potential customers to know my company uses its own products.  It is almost impossible to sell something if you cannot point to your own organization as "Customer Reference #1."

There is actually a Wikipedia page which defines this notion, also sometimes referred to as eating your own dog food or cooking.  It reads, "Eating your own dog food, also called dogfooding, occurs when an organization uses its own product. This can be a way for an organization to test its products in real-world usage. Hence dogfooding can act as quality control, and eventually a kind of testimonial advertising. Once in the market, dogfooding demonstrates confidence in the developers' own products."

I will give the MVP founder this credit: He has been able to convince his own junior high school to eat his own dog food.  In other words, they use MVP.  I know from writing to teachers at American Fork High School that they do not use MVP, except perhaps very rarely.  On July 23, 2019, I wrote to a Public Relations representative at Alpine School District to ask them these questions:

1. Does your district, which includes 9 high schools and 12 junior high schools according to your website, use the MVP math curriculum as a matter of policy?  
2. If not, then why not?
3. If curriculum decisions are not made at the district level but at the school level, can you tell me which schools of the 21 do use MVP as their math curriculum?

As of this publication date, I do not yet have an answer.  I do expect to receive an answer because this person has been helpful and prompt in the past.  Stay tuned here for an update.

So, here's issue #1 for me - before we even get to data: Why has MVP not been solidly adopted in Alpine School District and perhaps broadly across other parts of Utah?  I understand that this is not their product directly, but the State of Utah has had its hands in the original funding for MVP back in the 2012 timeframe, according to "STEM IS DEAD IN UTAH COURTESY OF THE USOE."  If MVP was widely adopted locally, then I believe we would have heard about it as Customer Reference #1.  But, we have no customer references for MVP other than the ones we have discovered on our own, or from teachers tweeting about attending MVP training.  MVP will not tell us who their customers are, other than to state they have customers in 30+ states.  Why the big secret? 

Here's issue #2 for me: Given the obvious lack of broad adoption of MVP in its home district of Alpine or its home state of Utah (or any other large districts in the country) why would "very large WCPSS" bite off on this venture with a very small company which lacks the staff and a proven ability to scale its product successfully? 

BACK TO THE DATA

In the prior article, I shared data for the 3 schools discussed above.  In this new chart, I am refining the chart to include additional important context including each district and the whole state.  Additionally, I am honing in on just 9th grade for the junior high data because in most cases, that is when Math 1 is taught, and that is where MVP is used at American Fork Junior High School.

This is an attempt to do a better job of convincing you that this data matters.  Specifically, we have 3 experimental groups:

1. American Fork Junior High 9th grade which has the best case scenario of MVP
2. American Fork High which receives the majority of its students from MVP-using American Fork Junior High
3. Fairfield Junior High 9th grade which does not use MVP, but instead creates its own curriculum

We also have several pseudo-control groups.  These are groups which, granted, do include the experimental groups as well, but the rest of the makeup of the control groups may or may not use MVP.  The assumption is that by and large, they do not.  Control groups are:

1. All Alpine District Junior Highs - 9th grade
2. All Alpine District Senior Highs
3. All Davis District Junior Highs - 9th grade
4. All Utah Junior Highs - 9th grade
5. All Utah Senior Highs

This is quite a busy chart, I know.  But please read it carefully to understand what it means.  I've used critical thinking to annotate it to assist with comprehension.

You can draw your own conclusions, but here is what I see:

1. American Fork Junior High (MVP Home) - fared slightly better (-8.7% decline in 3 year proficiency rate) than Alpine district as a whole (-9.3%). We don't yet know what curriculum the rest of the district is using but that is pending a response from Alpine.   
2. American Fork Junior High fared worse than Utah overall (+0.5% increase).  I see this as a RED FLAG because we have no indication that Utah at large is using MVP, so perhaps using MVP seems to be negatively impacting this one school disproportionately as compared to others in the state.  The MVP best case scenario should have been on par or better than the state if it truly is a superior curriculum.
3. American Fork Junior High fared considerable worse than a similar school Fairfield Junior High (+26.6%).  This was pointed out in the prior article but in that article I was looking at the blended scores of the junior highs, which would include Math 7 and Math 8.  So when I focus on 9th grade (which would be Math 1 and sometimes Math 2), the results are even more stunning for Fairfield Junior High!  I see this as a RED FLAG for MVP because the contrast is HUGE.
4. Fairfield fared quite better than other junior high schools in the Davis District (+11.2%).  Since Fairfield has reached higher proficiency levels, it may be hard to maintain the pace that the district makes.
5. Fairfield obviously is outpacing the state overall (+0.5%).
6. Davis District is clearly doing something well.  With the exception of Fairfield, they are using the Carnegie Curriculum.  My source tells me they are considering switching to more of a discovery learning model like MVP.  WHY. ON. EARTH??
7. American Fork High School is fed with MVP students and a few others from the district.  Their results (-32.3%) were noted as abysmal in the prior article.  The other high schools in the district are not faring too well either (-20.9%), but remember the terrible results from the high school are embedded in that 20.9% decline.  It takes more than one bad apple to spoil the whole bunch, girl, but this one is pretty bad.  I see this as a HUGE RED FLAG for MVP.
8. While all state high schools overall declined slightly (-4%), American Fork High School in comparison is way off the mark.  
9. Note that when comparing American Fork Junior High to Fairfield Junior High, Fairfield has a more challenging student population to educate in that it has a larger percentage of economically disadvantaged, students with disabilities, and English language learners.  So maybe we should be taking #equity curriculum advice from them instead of MVP.  YELLOW FLAG for MVP.

So let's recap those RED FLAGS comparing 3 year change in proficiency rates:

1. American Fork Junior High (-8.7) vs Utah (+0.5) = NET -9.2 WORSE
2. American Fork Junior High (-8.7) vs Fairfield Junior High (+26.6) = NET -35.3 WORSE
3. American Fork High (-32.3) vs Utah (-4) = NET -28.3 WORSE

To provide a little more detail, I did pull the actual test results for the 3 schools in question.  There was one anomaly in the data with the Math 3 scores for 2017.  The number of students assessed dropped considerably.  I saw this in other data in Utah as well, and have written them asking for an explanation.  So those data points are shown in red text below.

The table above color codes cells to attempt to show "cohorts" of students.  In other words, the yellow group takes Math 1 at American Fork Junior High in 2015, then Math 2 at the senior high in 2016, and Math 3 in 2017 (that is the suspicious data point).  The cyan and dark pink cohorts look more useful in that the number of students looks consistent.

You could probably find some other schools similar to American Fork Junior High which are not using MVP and which are doing worse.  I looked at this set of 3 schools due to one being the MVP home base (American Fork Junior High), and the other two having a data relationship with it: one based on student population progression (American Fork High) and the other on similarity (Fairfield Junior High).

I have no way of knowing the true cause of the steep decline in math proficiency at American Fork High School.  Is it the students coming in with 1 year (or 2 years, in the case of 68 students in 2016) of MVP in junior high school who lack foundations which impact their high school performance in Math 2 and 3?  Or is the Math 2 and Math 3 curriculum at the high school so bad is is disproportionately destroying these students' STEM potential?  Something is definitely wrong.

Alpine School District has a long history of trail-blazing experimental math programs such as ones based on "investigations math" or "constructivism."  It is right next the Brigham Young University, the source of at least one MVP founder. According to BYU's Math Education Department Fails Students, "BYU has for several years been the hotbed of new-age education fads.  Constructivism was pushed into Alpine School District thanks to the people at BYU."   In Investigations Math Summary, one parent gives background and plenty of resources.  Just search on the word "alpine."

I have tried to avoid confirmation biases I might have about this high school, but I can't help but wonder, which is my issue #3: I cannot for the life of me explain their steep decline in math proficiency other than to recognize that they are largely receiving students who went through MVP Math 1 (and sometimes Math 2) in junior high school.  Either that, or a major change at the high school that has gone uncorrected for 3 years.  My concern is that our WCPSS Math 1-2-3 MVP alumni will meet the same fate when they reach Trigonometry, Calculus, and/or college level math. 

I want to stress that this analysis is not a statistical analysis and article.  This is a common sense analysis and article.  It doesn't calculate margins of error and confidence levels.  I have no control over these experiments.  The samples which make up this data are not completely known other than what people have told me, and what I can infer by the data.  I am just a parent reading data available to the public, created in a state far away, about an experimental fad program which has been foisted upon tens of thousands of students per year and 50+ schools in Wake County, NC, much to our dissatisfaction.

You draw your own conclusions, and if you can find an MVP silver lining in this data, I'd like to hear it.