Then, the error for each of the fold is average,

where the model with the smallest amount of error is selected.

One major advantage to K-fold Cross Validation over Leave One Out Validation

is that it requires considerably less computational resources.

Because rather than fitting the statistical model as many times as

the number of observations in your data set, you fit it,

a substantially smaller number of times, typically less than 20 times.

Some statistical learning methods have computationally intensive fitting

procedures and data sets can have an extremely large number of observations.

This makes leave one out cross validation less feasible.

So, K-fold Cross Validation is a nice compromise between single data set

validation and leave one out cross validation.

In addition, K-fold Cross Validation often provides more

accurate estimates of the test error rate, than does leave one out cross validation.

Again, this has to do with the bias variance trade-off.

We know that the validation set approach can overestimate the test error rate,

because the training set will have only a proportion

of the number of observations in the full data set.

In the leave one out cross validation approach,

the training data set will have only one less observation than the full data set so

it can provide essentially unbiased estimates of the test error rate.

The leave one out cross validation approach is actually superior for

providing less biased estimates of the test error rate.

In the leave one out cross validation approach,

the training dataset will have only one less observation than the full dataset.

So it can provide essentially unbiased estimates of the error rate.

The leave one out cross validation approach is actually superior for

providing less biased estimates of the test error rate.

But bias is not the only thing we're concerned about.

We're also concerned about variance.

When it comes to having less variance in the test error rate,

the K-fold approach to the leave one out, cross validation approach.

Leave one, out cross validation has higher variance

than does K-fold Cross Validation.

This is because in leave one out cross validation,

the n-1 training data sets contain pretty much the same observations each time.

As a result, the estimates calculated in each cross-validation sample

will be highly correlated with each other, and

the mean of these highly correlated estimates will have greater variance.

With K-fold validation there's considerably less overlap

in the cross-validation samples,

which means less correlation between the cross-validation estimates.

And consequently, less variance.

For many statistical methods, cross validation is

easily conducted with procedures for functions that will do it automatically.

We just need to specify the type of cross validation.