Okay, hello everyone welcome. Now we've talked about some different techniques for assessing the quality of our data analysis. When it comes to design data, we're going to look over some examples of how to compute some of the measures that we were discussing within the R software. So back to our big picture in terms of the total data quality framework. Again, our focus right now is on best practices and evaluating the quality of the data analysis that we're performing in preparation for. Again, the reports, papers, presentations that would ultimately like to write or go over with our colleagues. We want to make sure that just because we've paid careful attention to quality along the other six dimensions of total data quality that we don't slide at this point of data analysis. We're going to see some examples of how to evaluate this with respect to design data within our right now. So we talked about this notion of variance inflation due to a weighted analysis of survey data. So recall that the use of survey weights and estimation when analyzing design. Survey data is designed to reduce bias in our estimates due to different sampling probabilities, survey nonresponse, etcetera. Generally a best practice to use those weights to make sure that we're fixing up some of those biases that often come with survey data. However, the use of weights and estimation can also inflate the variants of our weighted estimates affecting inferences like we talked about in the previous lecture. So the more variable that the weights are, the larger the inflation in the variants of the estimates will be, if the weights aren't correlated with our key variables using those weights may not be wise. So we can compute that diagnostic measure that we discussed in the last lecture to get a sense of the maximum potential inflation in the variants of a given estimate due to weighted estimation. If the weights are correlated with the key variables that we want to analyze, there may not be as much variance inflation, which is a good thing. This calculation assumes that the weights are not correlated with the variables of interest. And that's why the measure, the diagnostic measure that we're talking about here is considered an upper bound in terms of the variance inflation of our estimates. The higher the correlation of our weights with the variable of interest, the lower the variance inflation is going to be. So here's an example computation using the European social survey Russia survey data file. And along with this example you see an R Markdown file in the materials in addition to an html file that we've prepared, which shows the results of running that our Markdown file within our studio. So I'm going to flip over to that our R Markdown file here real quick and again, you can generate this html file that we're seeing on the screen right now. Simply by running the R Markdown file that we've provided with you to go along with these examples within our studios. You can open up that dot RMD file within our studio and then knit that Markdown file within our studio. You'll see a button for doing that and that will produce this kind of HTML file you can also knit to produce pdf files and word documents. In this case we knitted that our Markdown file to produce the HTML file that you see here. So the nice thing about our Markdown file is that you'll see the specific parts of the R code which are called chunks. Set aside from the rest of the output in the in the display here in those gray boxes like you see here. So the first R code we're reading in that ESS Russia dataset from that website that you see here. Then we're loading the package is associated with the tidyverse just to facilitate some data management here. Then we're loading the survey package enabling the analyses that were about to demonstrate. And then we get to our initial computation of this diagnostic measure. So remember this diagnostic measure for variance inflation due to waiting was taking the coefficient of variation of the weight squared and then adding one. So what this R code is doing here, you see, we're taking the standard deviation of the final survey weight in the Russia data frame. Okay, and we're dividing that by the meme of that final survey. Weight in the Russia data frame and then we're squaring that ratio, we're squaring the coefficient of variation and then we're adding one. So then in the output that you see here generated by our markdown you can see that that value is 1.347 roughly. So if we go back to the slides here are interpretation would be that at most the use of these survey weights and estimation will inflate the variance of a weighted estimate by 35% or 1.35 times. So this is a multiplicative inflation factor for the variants of our estimates. So this is an upper bound in terms of how much the variability in those survey weights is going to inflate a descriptive estimate based on the ESS Russia survey data set. So in general this type of variance inflation would be considered relatively minor. This is a quite reasonable value. If you see a one plus squared coefficient of variation value that gets to be like five or ten or something larger than that. That's suggesting that the variability in the weights may be overwhelming. Any reductions in bias in an estimate due to the use of weights and estimation and then using the weights and estimation maybe a bit questionable. Okay, so for larger values we would want to carefully assess how much our estimates are shifting when using the weights. Again calculating weighted estimates with the weights on and then comparing those two unweighted estimates with the weights not being used in your actual estimation. Another technique that we talked about for assessing the quality of a given analysis of design data is whether or not we need to use those survey weights when fitting regression models of interest, okay. So now we'll consider an example of formally testing whether accounting for survey weights is necessary when fitting a regression model to design survey data. When we're interested in looking at relationships between variables. So we're going to look at that example in the R Markdown code and that html output that we were looking at from today for the details of this kind of analysis, all right. So let's turn back to our html file. Okay, and another nice thing about our Markdown that you can see here is that you can intersperse notes and information that others can read about the analysis that you're conducting. So you can mix the R code, these chunks of our code with actual text that explains what you're seeing in the our output. And you can see that in the RMD file that we've provided for you to open within our studio, okay. So we're going to see how to test the need for using survey weights when we fit a regression model. And the first thing that we're going to do this first chunk of our code here is to create an index of overall satisfaction with life in the ESS Russia dataset. So to do that we're actually using the mutate function which is part of the tidyverse. A group of functions that can be used for data management, okay. So what this code is saying is that we're taking the Russia data frame object and then you see what's called a piping operator that percent and then the greater than sign and then the percent that says input. That Russia data frame object into the next sequence of functions or commands. And that next function is called mutate. So we're submitting that Russia data frame object to this mutate operation. And what we're doing is we're creating a new variable called overall set. Which is the sum of the variable STF life, STF echo and STF gov like in a measure of overall satisfaction in the ESS Russia survey data set. And then we're storing the results of that operation in the data frame object called Russia. So basically we're overwriting the Russia data frame with the results of this operation adding overall set to our Russia dataset. Next what we're going to do is declare the key features of the European social survey sample design. Those final survey weights, those stratum codes that reflect the sampling procedure used and those cluster codes that reflect the sampling procedure used. Remember where the cluster sampling tends to inflate the standard areas of our estimates. We're going to declare all of those key design features of the sample design in a new design objects called Russia dot DSGM. And this is using a function that's available in that survey package within our that we just loaded. This is just to declare the features of this sample design so that we can account for those features when we're actually analyzing the survey data from Russia. Once we declare that design object and you see that we're doing that here, we're using the SVY design function. Within the survey package, we're declaring those PSU codes as this IDs argument. That's where we indicate the PSU or the cluster codes. Were are declaring the stratum codes which are in the variable called stratify to the strata argument after until day. The data frame object is called Russia. And we're using that data frame object where we just added that overall set variable. We're saying that those sampled clusters are nested within the strata, which is common in these types of sample designs. And then here's where we declare our final waits there. The weights argument is saying PSP wait our final weights. And after we declare this object called Russia dot DSGM. We can then look at a descriptive summary of the design information that we just specified. And so in this html file that was generated by running or knitting that dot RMD file within our studio. You can see that overall summary of the design here. You see how many clusters we have. We see eight different strata indicate in the in the ESS Russia sample design and then the names of the variables in the actual data set that we're working with. So there's a lot of information containing that Russia dot DSG and design object. And we're going to keep referring to that design object when we go through this analysis. Okay, so the first thing that we're going to do is fit our regression model to the ESS Russia survey data set fully accounting for those design features. And to do that we use the SVYGLM function contained within the survey package that we loaded. So this is another function that's contained within that survey package that we we loaded with that require statement earlier. So in SVYGLM, we're saying that our dependent variable of interest is overall set that variable we just created on the left hand side of the till day. And our predictors are going to be the age of the survey respondent. The marital status of the survey respondent which is declared to be categorical in this regression model. We do that using this factor function and then this indicator variable of whether or not the respondent was a male or one, zero variable. So those are three predictors of overall satisfaction with life. And then after we specify the model, we indicate the name of that design object. We just created that design object contains the data and it contains those design features, the weights in the survey data set. The stratum codes and the cluster codes for this kind of analysis, we make sure that we're fully accounting for all those features including the weights. So we can use this SVYGLM function to fit the model. And then we're going to save the results in this new object called fit.surv. And then we can use the summary function with that fit.surv object to look at the estimates in our regression model and that's the next set of output that you see here in the html file. Okay, so these are the estimated coefficients for our target population in the based on the ESS Russia survey data set. Because we use the weights in estimation. So you see our estimated intercept in the regression model and then the estimates of the coefficients for all the other predictors that we included in this model. And we also see indications of whether or not those coefficients are significantly different from zero. So this approach that we just went through yields design unbiased. So based on the sample design that was used for the ESS. These are unbiased with respect to that sample design estimates of the regression coefficients due to the fact that we're using weights to fit this model in addition to their standard errors. You can see those standard errors in the standard error column here. This is generally best practice. Again, fully accounting for all these design features used for the sampling in estimating our coefficients of interest. However like we talked about in lecture when we fit these regression models to design survey data. The use of weights and estimation may not be necessary. If we've done a good job specifying that model and we've captured these relationships between age and marital status and male in the target population, okay. In these cases the use of weights may simply inflate the variances of our estimated coefficients. And this makes it harder to test the coefficients for significance and this tends to reduce our statistical power. So we can test for the need to use weights by declaring a new design object which ignores those weights and estimation. Okay, so notice now we're again using that SVY design function. Same cluster codes, same stratum codes defining our sample design. But now instead of declaring the weights variable, we're just declaring a one after the till day. And that means that everybody in the dataset just gets a weight of one instead of the weights that were in that variable to begin with. Okay, everything else is specified as the same. We still need to account for the cluster sampling and the stratification in terms of our overall variance estimation. But I'm saving this new design object in a new object here called russia.dsgn.nowgt. So to indicate that this is a different design object where the weights have been ignored. Okay, then I can look at a summary of that no wait design object and you can see that are interpreting all of our probabilities of selection as being one. Okay, this is an equal probability of selection design where waiting is not necessary because those weights are being ignored. Everything else is the same, okay. So then what we do is we refit our model of interest ignoring the weights. So this command looks exactly the same SVYGLM function. But now we're using that different design object russia.dsgn.noweight. And we're saving the results of that model. Fit in a new object called fit.surv.noweight. And then we're applying the summary function to that new object here. So now we see our new set of output, the estimated coefficients, their standard errors and again tests of significance. And looking at this output. We see that the estimates and our conclusions about the relationships based on these estimates would be quite similar to those based on the fully weighted analysis. The coefficients look very similar. So the question is, is the use of weights and estimation, telling us anything about our coefficients of interest or is the use of weights only inflating those standard errors. So to formally test the need for the weights, we add the final weight variable as a predictor to the unweighted model. So noticing the model specification, we're now adding the weight as a predictor and then we're also adding these two way interactions between the final weight variable and each of the other predictors. So the weight times age, the wait times the categorical marital status factor and the wait times male. Okay, same specification. Otherwise we're just adding the weight and these two way interactions to that full model. And again, we're using the no wait or unweighted design object here to fit that model. Okay, then after we fit that model we perform a wall test and we can do that using red term test which I'll come back to in a second. Okay, but here are the estimated coefficients in this model adding those predictors representing the weight. And then the interactions of the weight with each of the other predictors that were really interested in. So if any of these new coefficients were significant, that would suggest that the weight is either changing the value of the intercept. Okay, if this weight coefficient is significant or that the weight is changing the relationships of our predictors of interest with the outcome. Which means again that the weight is informative about our coefficients of interest. And looking at these p values here you see that none of these additional coefficients involving the weights really seem like they're significant. Okay, so those ratios of the coefficients to their standard errors are all pretty large. So we can do that overall wall test for those additional terms that we added to the regression model by using this reg term test function which is also available in the survey package. So to use reg term test, we take our model fit objects which included those additional weight predictors and after a comma. And until day we simply indicate the names of those additional predictors, okay. The weight, the interaction of the weight with age using a colon. The interaction of marital status with the weight using a colon and then the interaction of the weight with male, okay. That will give us an overall test of the null hypothesis that the coefficients of those additional weight terms are all equal to zero. Mean that the weights are unnecessary. Okay, so here are the results of that wall test and you can see that the p value for that wall test is about 0.253. So what that means is that we would fail to reject the null hypothesis that all those coefficients are different from zero. And what that means is that the weights aren't changing the estimates of the coefficients in our model. All they're doing is inflating the variants of those different estimates. All right, so the use of the weights and the analysis in this particular example does not seem to be necessary if that p value is really small close to zero. Less than .05 say, then that suggests that the weights are in fact moderating some of our coefficients in this particular model. And we would probably want to keep the weights and account for the weights in our estimation because they are informative about the regression coefficients of interest. Okay, so in this example it seems like we've done a good job of specifying our regression model. We've captured the relationships well, and we don't really need to account for those weights in the overall estimation. Okay, so walking through that example, in our Markdown, we see that the use of weights was not necessary when fitting this particular model. And this is always a good test to consider to make sure that you're not computing inefficient estimates, estimates with with too much standard error. Again, due to the variability in the weights. Okay, so now that we've seen examples at looking at these measures of the analogy of quality in terms of the analysis of design data. We're going to turn to a discussion of measuring data analysis quality for gathered data. And in that discussion we'll see additional examples of computing data analysis, quality metrics for gathered data as well that we can think about. So thank you.