So now we can take a look at the competition itself. This is just a timeline from 2006 until ending around June/July 2008. So it started again in October 2006 and in fact within a week of the start of competition CineMatch is beaten by 1%. So one team had actually gotten 1.06% improvement. So in this graph right here we , the actual values that are plotted the 0.9514 that was CineMatch's original value. This 0.9413 is where they were a week after. And that's a 1.06% improvement over 0.9514. So we're referencing this all to 0.9514, so this one's 8.26% better than this. This one is 8.43% better than this, just to give an idea. Now, at the end of each year, Netflix had what were called Progress Prizes, and it would, it would award a certain amount of money. I think it was $50,000 to whichever team had gotten the, the most improvement over that year, basically. And there were only. O-, one of the only, the only requirement for that was it had to be at least 1% over the previous year. So in order for you to qualify for the Progress Prize your algorithm had to beat whatever was in the Progress Prize the previous year by at least 1% and it didn't it only went on for you know two years. But so the 2007 Progress Prize was won by this team called BellKor and BellKor had reached a 8.43% improvement. Right. So, you see already. By 2007, they had already reached 8.43% improvement in the RMSE and this was about a week before they, final call for the Progress Prize. So, what happened was they would say okay, well, last call you have 30 days to make your final submissions and. And then at the end of that then they would run and see who was doing the best. And at that point, BellKor had had an 8.43% improvement. And that was the best one so they won that Progress Prize. So you can see we're already over 8% improved over CineMatch. So setting 10% as the target goal. They didn't necessarily notice, they probably had a really good idea. But 10% was, a really good target and you can see clearly because if they hadn't done that then within one year, if it was like 8% or 5% or something, someone would have already beaten it. But it still took an extra year on top of that before they actually could beat and get the amount of improvement. So in 2008 the Progress Prize was won by the team called BellKor in BigChaos, and so it was actually BellKor which merged with another team called BigChaos. And then they form this super team, so to speak, and they got 9.44% improvement. So, one thing, again, that you'll notice is that we jumped from 1% to 8.43%, from 2006 to 2007 then 2007 to 2008 we only improved by like another percent. So you see that the last few percentages are really hard to squeeze out in that. And this actually determined that 2009 was going to be the last year, because remember in every year you had to improve it by at least 1% over the previous year, so they needed to get at least 10% by the next year. So now the, the grand finale and this is pretty interesting and funny story actually. There's this other team called the Ensemble, which reached 10.10 on the quiz set, so they. BellKor, it was actually by this time called BellKor's Pragmatic Chaos, because they merged with another team called Pragmatic Theory. They had gotten 10.05%. then they enter the last call phase. Then the ensemble had gotten 10.10% on the quiz set by the end of the last call phase, so it's just slightly better, right. And, then they entered the, the final call, where you just had to just submit your algorithms and, it turns out that the ensemble and BellKor's Pragmatic Chaos both got 10.06%, 'kay. But BellKor's Pragmatic Chaos submitted their algorithm 20 minutes earlier. So this is again, this is on the quiz set and in fact the Ensemble had a slightly better performance on the quiz set than BellKor's Pragmatic Chaos did. But they both tied at 10.06% on the test set. But BellKor's Pragmatic Chaos submitted the algorithm 20 minutes earlier, as so they were the winners of the million dollar prize and the Ensemble lost for that reason and so a very exciting 24 month period came down to a 20 minute differential.
