Different types of uncertainty are associated with different types of problem-solving and decision-making processes. This was the key message with which Arnaldo ended our previous session. In this video, we're going to focus on different types of errors that can occur in decision-making processes, and their implications for innovation and performance. Error types are an essential aspect of decision-making, especially under conditions of uncertainty, and are going to be essential when we talk about hypothesis testing, and A/B testing in particular. As much as we might all be familiar with errors, they remain an abstract concept. So, to fix ideas, let's focus on a recent example before looking at key definitions. You might have heard of Theranos, a startup founded in 2003 by Elizabeth Holmes. Theranos promised a new era in medical diagnosis when a single finger prick producing one drop of blood could yield a multitude of diagnostic tests: from hematology to general chemistry, to even molecular diagnostics. The key selling point of Theranos was the promise of faster and more convenient and cheaper results for patients, and overall the dream to reduce the emotional and physical trauma that for many is associated with a needlestick. The product was introduced in Walgreens across the US in 2013, but was eventually recalled after a Wall Street Journal article investigated and showed that blood tests and devices by Theranos gave inaccurate results. Official investigations followed, and in 2016 Walgreens ended its relationship with Theranos and closed its in-store blood collection centers. So, how did Theranos invest time and resources in developing a faulty product? Well, the company conducted some clinical trials when they signed to introduce their blood testing products and faced the possible scenarios represented in a two-by-two matrix that you're seeing on the screen. As you can see, there are two correct options. Their product is effective and should be produced and released, or their product is not effective and should not be produced and released. There is also room - in the top right and bottom left corners - for error. What happened in Theranos case was that the product was not effective. However, a series of conducted tests showed that the products could work and the company decided to produce it and release it. This is what we call a false positive, or type I error. A type I error is where you expect a positive results, so - in this case - the product to work, but in fact, the result is negative. This is why we call this type of error a false positive. Now, let's assume - instead - that the company had a product that was working, but preliminary lab tests incorrectly suggests that it was not effective. In this case, had Theranos not produced the product based on this information, they would have incurred in the error represented the top right corner of the matrix, or a type II error. A type II error occurs where you expect a negative result: the product doesn't work, but the result is actually positive. And that is what we call a false negative. The key reason why we differentiate these errors is because they have very different consequences for innovation and performance. In the case of Theranos, their false positive error resulted in inaccurate and faulty diagnoses of several conditions, aside from catastrophic financial consequences. A type II error, instead, would have resulted in the firm not making profits by not releasing their product. Research in this area has investigated type I and type II errors at various phases of new product development, in several industries and contexts. In the case of Theranos, thorough lab tests could have objectively revealed the viability of the products. In other contexts, instead, this is not always an option. Work by Kimberly Alice Buck from UC Davis School of Management and Roderick Cromer from Stanford in particular, try to understand why so many organizations incur in false negatives by rejecting great ideas. Alice Buck and Cromer studied Hollywood pitch meetings in which screenwriters present their movies and TV shows ideas to executives that decide whether to produce these shows or not. As Alice Buck and Cromer explain, Hollywood executives often commit false negative errors, passing on scripts that proved to be extremely successful when someone else takes the chance and produces them. For example, the scripts for Star Wars, Titanic and The Truman Show were all rejected several times. Hollywood executives seem to develop fairly elaborate criteria for forecasting the success of screenwriters in the scripts. But overall, we lack evidence of whether these criteria are actually useful for accurately predicting success. Executives may be rejecting great ideas without ever knowing it. Especially if another executive does not produce scripts that they reject. Justin Berg from Stanford tries to understand what goes wrong in the process of evaluation of creative ideas in the context of Cirque du Soleil productions. He finds that those who create content for the Cirque shows are skilled at predicting the success of Cirque shows created by others, but not the shows that they create themselves. And interestingly, content creators in this context are more accurate in predicting success than executives who decide which Cirque shows should be produced. Taken together, these studies show that in contexts where it's harder to find objective evidence of the variability of the idea - and not everyone can in fact conduct laboratory tests - understanding what type of error I is facing might be even more difficult. This is also the case in the context of entrepreneurship, where startups often face a decision to either pursuing your business idea or not. And we use the Inkdome case from the previous lesson. In this case, the two-by-two matrix looks something like this. So, either the idea to create a search engine for a tattoo artist is profitable or not, and either the founders pursue it or abandon it. So, how would you place errors and correct choices in this matrix? Take one minute to think about the correct answers. If we set aside the two obviously correct choices, so either the search engine idea for a tattoo artist is profitable and the founders correctly pursue it, or the idea is not profitable and founders do not pursue it, we can see the founders might be making two errors. They might be working on an idea that is not profitable, yet they decide to pursue it, which will be a false positive, or they might decide not to pursue a profitable idea, which is a false negative. Well, we're not trying to outline which situation might be better than the other. The key point is that it is difficult for an entrepreneur to understand which scenario he or she is facing. Very much like in the example of the creative arts. As a consequence, entrepreneurs are facing a situation where there is more room for error. The key takeaway from this lesson is that while every decision leaves room for error, the likelihood to commit an error increases as the uncertainty of the context in which we operate increases. Before focusing on how scientific approach decision-making can reduce the occurrence of these errors, we will examine the errors other companies have done to better understand false positives and false negatives.
