Now we extend our discussion on categorical variables,from working with only one categorical variable at a time, to working with two categorical variables, and evaluating the relationship between them. So in this case, we're going to label one of our categorical variables the explanatory variable, or the grouping variable. And the other one, our response variable. And so, were working with categorical variables that have two levels, so a success and a failure is what were going to be categorizing them as. Were going to calculate proportion of successes in the two groups ,based on our sample, and were going to compare this to each other to be able to say something about how the proportion of successes, in the population compared to each other. And since we're calling this an estimation, as usual we're going to use a confidence interval. In other words, what we're going to do in this video could be summed as, calculating a confidence interval for the difference between the two population proportions ,that are unknown using data from our sample. Here is the data set that we will be working with. In early October 2013, a Gallup poll asked, "Do you think there should or should not be a law that would ban the possession of handguns, except by the police and other authorized persons?" The possible responses on the survey were no, there should not be such a law, yes, there should be such a law, or no opinion. And we're going to categorize ,yes there should be such a law as success, and we're going to categorize either no opinion ,or no there should not be such a law as failure. Let's take a look at the survey results from Gallop first. Gallop actually has been asking this question. On support for ban on possession of hand guns, since 1959. And we can [BLANK_AUDIO] Look to see how things have changed in the U.S. since then. The dark green line represents the percentage of respondents saying, no there should not be such a law. And the light green ,is the percentage of respondents that say, yes there should be such a law. So back in the 1950s or early 1960s, the case was that there were higher proportion of people thinking yes ,there should be such a law and, today that's a much lower proportion at 25%. Compared to the 74 that say no there should not be such a law. 74 plus 25 makes 99% so that 1% missing must have been the proportion of people who said they had no opinion on the matter. We also asked this question to a group of Coursera students as well, and here is what the responses looked like. Of the 83 responses we got [INAUDIBLE], 59 people, so 71%, said yes, there should be such a law, and 24%, 20 out of 83, said no there should not be such a law, and 4 out of 83 said that they had no opinion. This seems like a very different population and that's probably expected. Coz there is a much more international population than the US and ,possession of guns and what not tends to be a hot topic in the US, may not necessarily be so in other countries. So this is what we're seeing as a distribution. We can actually take a look at our results summary. So in the U.S. Gallup had surveyed 1,028 people and 25% of them said yes there should be such a law. That makes 257 people. And within the Coursera population we had a sample of 83 respondents. And the proportion of success, so [INAUDIBLE] respondents that said yes there should be such a law is 71%, so that's 59 out of 83. Using these values, we want to answer the following question, how do Coursera students and the American public at large compare with respect to their views on laws banning possession of handguns? So our parameter of interest is the difference between the proportions of all Coursera students ,and all Americans who believe there should be a ban on possession of handguns. So that could be, P of Coursera minus P of U.S. P stands for the population proportion, or unknown population parameter. And our sample statistic, that serves as a aa point estimate for this parameter of interest, is the difference between the proportions of sampled Coursera students and sampled Americans ,who believe there should be a ban on possession of handguns. And that if, that could be denoted as P hat of Coursera, minus P hat of US. So how do we estimate differences between proportions? Well, estimation we said is usually a confidence interval. And a confidence interval is always of the form point estimate plus or minus a margin of error. In this case our point estimate ,is the difference between two sample proportions ,and our margin of error as usual, can be calculated as a critical score, this is a critical Z score, we're dealing with sample proportions ,and based on the central limit theorem, we know that the distribution of those are going to be nearly normal ,for the difference between the sample proportions as well, so we're using a Z score, for our critical score, times we have the standard error of the difference between the two sample proportions. So once again, the new concept here is, how do we calculate the standard error ,for the difference between two proportions, for calculating a confidence interval . We're going to get to in a little bit why we're, you know, specifying for calculating a confidence interval. But ,we've already seen this before with the one proportion case, as well. So, there's going to be a difference between how we calculate this for confidence interval versus hypothesis tests. This video focuses on confidence intervals. The next one will walk you through how to do this as [INAUDIBLE] a hypothesis test. So the standard error ,for the difference between the two sample proportions ,can be calculated as the square root of P hat 1. That's the proportion of successes in group one ,times 1 minus P hat 1, that's the proportion of failures in group 1. Divided by that group sample ,size and then the same quantity for the second group as well. Important thing to mention here are that these are P hats ,so observed sample proportions. We add up these two quantities ,just like working with means, we're bringing two unknowns together, a P hat one and a P hat two ,sample proportions from two groups ,so the overall variability should increase and hence we're adding the two Variability components here, and then finally taking the square root of that, to go from variance, to standard deviation, or in other words, the standard error, which is basically the standard deviation of the sampling distribution. Of course, we just introduced a new formula, that means we need to go through the conditions as well. The first condition, as usual, is independence. Within the groups, we want to make sure that sampled observations are independent of each other. And we can ensure this using a random sample or assignment, depending on whether we have an observational study or an experiment ,and if we are sampling without replacement, we want to have our sample size be less than 10% of our population. And between groups, we want to make sure that the two groups are independent of each other. In other words, the data are not paired. From means, we had actually introduced a method for working with paired means, but for proportions, we're only going to focus on ,methods that work with Independent groups for comparing two proportions. We also need some sample size conditions ,to ensure normality of the sampling distribution ,and this is that each sample should meet the success-failure condition. So we want to meet the success-failure condition for the first sample as well as the second sample. And remember that if we're doing a confidence interval, these proportions that we're considering to calculate the success-failure condition need to be our observed sample proportions. So, let's put all of that to use. Using a 95% confidence interval, let's estimate ,how Coursera students and the American public at large ,compare with respect to their views on laws ,banning possession of handguns. And we have our data here as well, but before we can move on to do doing the confidence interval, we need to check our conditions. Independence condition. We do have a random sample ,for the US population, but not for the Coursera population. This was simply a voluntary poll that was posted on the Coursera discussion forums, so we can't say that this is a random sample. So, while Gallup does a good job of getting a random sample from the US, we did not, certainly did not do so for the Coursera students. And for the 10% condition though, that's math for both of the groups, because both 1028 and 83 are less than 10% of their respective populations. What does that mean? The sampled Americans ,can be assumed to be independent of each other. However the sampled Courserians may not be. So in this case, we want to be kind of wary of [INAUDIBLE] generalizing any of our conclusion from these findings ,to the overall population at large ,because we don't really have a good sample from the Coursera population. We're still going to move on with our analysis only for illustrative purposes those. So, in this case because we don't have a good sample, we want to make sure that we're being very very careful about generalization at the end of the confidence interval calculation. The other condition we need to meet ,is the sample size and skew condition. So this is about, whether we can assume the sampling distribution of the difference between the two proportions to ,be nearly normal. And when we're dealing with proportions ,we check for this using the success failure rule. And when we're doing a confidence interval we check for this using the observe successes and observe failures. In the US ,we had 257 observed successes and 1,028 minus 257, 771 observed failures. Here what we mean by observed successes ,is those who voted for yes, there should be a ban on possession of handguns, and the remainder are those who said no, there should not be or no opinion. For the Coursera students ,we have 59 successes ,and 24 failures, so it appears that both of these are met, the suc success failure condition is met. For both of the groups and therefore we can assume. That the sampling distribution ,of the difference between ,two proportions is nearly normal. Now that we got our conditions out of the way. We can actually calculate our confidence interval. That's a point estimate so the difference between two sample proportions. Plus or minus some margin of error, that we calculate as a Z star, times the standard error. The sample proportion for Coursera students is .71, and the sample proportion for Americans is .25, so, our point estimate is the difference between them, since we are asked for a 95% confidence interval, the Z star is 1.96, and we can calculate our standard ,error as the square root of the sum of two quantities. So, for the first quantity, we're looking at the proportion of successes in Coursera times the proportion of failures divided by the sample size for the Coursera students ,plus the proportion of the successes in the US times the proportion of failures divided by the sample size for the US. [INAUDIBLE]. The standard error ,then comes out to be .0516 and based on this, the margin of error comes out to be roughly 10%. In this case, our overall confidence interval comes out to be between .36 and .56. What this means is that we are 95% confident ,that the proportion of Coursera students who believe that their should be a ban on possession of handguns ,is 36 to 56% higher than the proportion of Americans who do. That's a huge difference even when we factor in the variability ,around the point estimate. And, again, this is probably expected based on how different the composition of the two populations are. So, we were asked to estimate the difference ,and we made an executive decision to put the Coursera student proportion First, and the American proportion second. But does the order actually matter? So does our decision actually change things? The answer is yes and no. It might change some of the calculations along the way, but it's not going to change your conclusions. Remember, this is how we calculate our confidence interval for the difference between two proportions. The point estimate ,plus or minus a margin of error. Your margin of error is bound to be a number that's always, always positive, because if you think about it, the standard error is always going to be positive. Conceptually speaking negative variability doesn't make sense, and mathematically speaking, the standard error is a quantity that is calculated as a square root. And your Z star by definition is always going to be positive as well. On the other hand, your point estimate can be positive or negative. Depending on in which order you're subtracting things. In our calculations, we chose to put Coursera first ,and US second. So we were looking for the difference between the proportions of Courserians, all of them, and Americans. All of them who believe that possession of hand guns should be banned. And because we don't know these true proportions, we decided to estimate this using a confidence interval. In construction of our confidence interval, since we put Coursera first, we first plugged in the sample proportion for that group, subtracted from that the sample proportion for Americans. Added and subtracted the margin of error. This gave us .46 plus or minus .10 which yielded a confidence interval of .36 to .56. How do we reverse things though? The second line of the calculation would be only slightly different. It doesn't matter in which order you calculate the standard error. You would still get the exact same standard error ,and hence the exact same margin of error. So, the .10 stays, but the point estimate ,would now be calculated as .25 minus .71, which would yield negative .46 plus or minus .10, which is going to yield what looks like a slightly different confidence interval. So, this goes from a negative .56 to negative .36. Well, I think we can see a pattern here. What we're seeing here is that ,the bounds of the confidence interval are now flipped, and their signs are flipped as well. So, instead of .36 to begin with, we have negative .36 to end with, and instead of .56 to begin with, we have negative .56 to end with. So, what's going on here? Well, these are actually exactly the same thing. The first interval could be interpreted as we are 95% confident that the proportion of Coursera students ,who believe there should be a ban on handguns is ,36% higher to 56% higher than the proportion of Americans who do. Or we could rephrase the same thing as the proportion of Americans. Who believe there should be a ban on han, possession of handguns is, 56% to 36% lower, than the proportions of Courserean's. So, it's just which one you put first and which one you put second in your interpretation, but as long as your interpreting correctly ,it doesn't matter in which order you do the calculations. We're still going to get the same thing, except we're going to make up for it in our interpretations. Now, one last step working with this confidence interval. Based on the confidence interval we calculated, should we expect to find a significant difference at the equivalent significance level. Between the population proportions of Coursera students and the American public at large ,who believe there should be a law banning the possession of handguns. And remember our confidence interval was from .36 to .56. What we're asked to do here is kind of a mock hypothesis test, and in that case, our null hypothesis would be That the difference between the two population proportions is 0, because that basically says that no, there's nothing going on. These two populations are exactly the same. This is what our confidence interval would look like on a number line. Anything between .36 and .56 is fair game ,for a difference between the two population proportions. In that case ,the value zero ,for the difference between the two population proportions does not appear to be fair game and ,based on that we would actually reject this null hypothesis ,and say that based on this confidence interval. It doesn't appear that these two populations are the same with respect to proportion of those who believe there should be a law banning the possession of handguns.
