Introduction

Resampling methods involve repeatedly drawing samples from a training set and refitting a model of interest on each sample in order to obtain additional information about the fitted model. (e.g. cross-validation, bootstrap)

• Estimates of test-set prediction error (CV)
• S.E. and bias of estimated parameters (Bootstrap)
• C.I. of target parameter (Bootstrap)

Cross-Validation

The training error rate often is quite different from the test error rate, and in particular the former can dramatically underestimate the latter.

• Model Complexity Low: High bias, Low variance

• Model Complexity High: Low bias, High variance

Prediction Error Estimates

• Large test set
• Mathematical adjustment
  • $C_p=\frac{1}{n}(SS{E}_d+2d{\hat{\sigma }}^2)$
  • $AIC=\frac{1}{n{\hat{\sigma }}^2}(SS{E}_d+2d{\hat{\sigma }}^2)$
  • $BIC=\frac{1}{n{\hat{\sigma }}^2}(SS{E}_d+log(n)d{\hat{\sigma }}^2)$
• CV: Consider a class of methods that estimate the test error rate by holding out a subset of the training observations from the fitting process, and then applying the statistical learning method to those held out observations.

The Validation Set Approach

A random splitting into two halves: left part is training set, right part is validation set.

Drawbacks

• The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
• Only a subset of the observations are used to fit the model.
• Validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.

Leave-One-Out Cross-Validation

LOOCV involves splitting the set of observations into two parts. However, instead of creating two subsets of comparable size, a single observation $(x_1,y_1)$ is used for the validation set, and the remaining observations $(x_2,y_2),...,(x_n,y_n)$ make up the training set.

In Linear Regression

$$C{V}_{(n)}=\frac{1}{n}\sum _{i=1}^n(\frac{y_i-\widehat{y_i}}{1-h_i}{)}^2$$

• $C{V}_n$ bacomes a weighted MSE

Drawbacks

• Estimates from each fold are highly correlated and hence their average can have high variance.

K-fold Cross-Validation

This approach involves randomly dividing the set of observations into k groups, or folds, of approximately equal size. The first fold is treated as a validation set, and the method is fit on the remaining k − 1 folds. This procedure is repeated k times; each time, a different group of observations is treated as a validation set. This process results in k estimates of the test error. The k-fold CV estimate is computed by averaging these values. If k=n, then it is LOOCV.
$$\begin{gathered} C{V}_{(k)}=\frac{1}{k}\sum _{i=1}^{k}MSE \\ or \\ C{V}_{(k)}=\frac{1}{k}\sum _{i=1}^{k}Er{r}_k \end{gathered}$$
Typically, given these considerations, one performs k-fold cross-validation using k = 5 or k = 10, as these values have been shown empirically to yield test error rate estimates that suffer neither from excessively high bias nor from very high variance.

Bootstrap

• A powerful statistical tool to quantify the uncertainty associated with a given estimator or statistical learning method.
• For example, it can provide an estimate of the standard error of a coefficient, or a confidence interval for that coefficient.

Steps

• Obtain datasets ($n$ observations) by repeatedly sampling from the original data set $Z$ with replacement $B$ times.

• Each of these bootstrap data, denoted as $Z^{\ast 1},...,Z^{\ast B}$, is the same size as original dataset $n$. And bootstrap estimates for $\alpha$ denoted as ${\hat{\alpha }}^{\ast 1},...,{\hat{\alpha }}^{\ast B}$. Thus some observations may appear more than once and some not at all (2/3 of original dataset).

Estimate of S.E.

$$S{E}_B(\hat{\theta })=\sqrt{\frac{1}{B-1}\sum _{r=1}^B({\hat{\theta }}^{\ast r}-{\bar{\theta }}^{\ast }{)}^2}$$

Estimate of C.I.

Bootstrap Percentile C.I.

$$[L,U]=[{\hat{\theta }}_{\alpha /2}^{\ast },{\hat{\theta }}_{1-\alpha /2}^{\ast }]$$

Bootstrap S.E. based C.I.

$$[L,U]=\bar{\theta }\pm z_{1-\alpha /2}\times \frac{S{E}^{\ast }}{B}$$

Better Option (Basic Bootstrap/Reverse Percentile Interval)

$$[L,U]=[2\hat{\theta }-{\theta }_{1-\alpha /2}^{\ast },2\hat{\theta }-{\theta }_{\alpha /2}^{\ast }]$$

Key: the behavior of ${\hat{\theta }}^{\ast }-\hat{\theta }$ is approximately the same as the behavior of $\hat{\theta }-\theta$.

Therefore:
$$\begin{gathered} 0.95=P({\hat{\theta }}_{\alpha /2}^{\ast }\le {\hat{\theta }}^{\ast }\le {\hat{\theta }}_{1-\alpha /2}^{\ast }) \\ =P({\hat{\theta }}_{\alpha /2}^{\ast }-\hat{\theta }\le {\hat{\theta }}^{\ast }-\hat{\theta }\le {\hat{\theta }}_{1-\alpha /2}^{\ast }-\hat{\theta }) \\ =P({\hat{\theta }}_{\alpha /2}^{\ast }-\hat{\theta }\le {\hat{\theta }}^{\ast }-\theta \le {\hat{\theta }}_{1-\alpha /2}^{\ast }-\hat{\theta }) \\ =P({\hat{\theta }}_{\alpha /2}^{\ast }-\hat{\theta }\le \hat{\theta }-\theta \le {\hat{\theta }}_{1-\alpha /2}^{\ast }-\hat{\theta }) \\ =P(2\hat{\theta }-{\theta }_{1-\alpha /2}^{\ast }\le \theta \le 2\hat{\theta }-{\theta }_{\alpha /2}^{\ast }) \end{gathered}$$

In General

• Each bootstrap sample has significant overlap with the original data. This will cause the bootstrap to seriously underestimate the true prediction error.

  • Can partly fix this problem by only using predictions for those observations that did not ( by chance ) occur in the current bootstrap sample. (Complicated)

• If the data is a time series, we can’t simply sample the observations with replacement. We can instead create blocks of consecutive observations, and samp le those with replacements. Then we paste to gether sampled blocks to obtain a bootstrap samples.

Bootstrap in Regression

$$Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i,i=1,...,n$$

Find S.E. and C.I. for ${\beta }_0$ and ${\beta }_1$

Empirical Bootstrap

• Resampling $(X_1,Y_1),...,(X_n,Y_n)$ and obtain:

  • Bootstrap sample 1: $(X_1^{\ast 1},Y_1^{\ast 1}),...,(X_n^{\ast 1},Y_n^{\ast 1})$
  • Bootstrap sample 2: $(X_1^{\ast 2},Y_1^{\ast 2}),...,(X_n^{\ast 2},Y_n^{\ast 2})$
  • …
  • Bootstrap sample 1: $(X_1^{\ast B},Y_1^{\ast B}),...,(X_n^{\ast B},Y_n^{\ast B})$

• For each Bootstrap sample, fit regression and obtain $({\hat{\beta }}_0^{\ast 1},{\hat{\beta }}_1^{\ast 1})...({\hat{\beta }}_0^{\ast B},{\hat{\beta }}_1^{\ast B})$, then estimate S.E. and C.I.

Residual Bootstrap

• Recall that residuals to mimic the role of $ϵ$.

• Bootstrap the residuals and obtain:

  • Bootstrap residual 1: ${\hat{e}}_1^{\ast 1},...,{\hat{e}}_n^{\ast 1}$
  • Bootstrap residual 1: ${\hat{e}}_1^{\ast 2},...,{\hat{e}}_n^{\ast 2}$
  • …
  • Bootstrap residual 1: ${\hat{e}}_1^{\ast B},...,{\hat{e}}_n^{\ast B}$

• Generate new bootstrap sample: $X_i^{\ast b}=X_i,Y_i^{\ast b}={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i+{\hat{e}}_i^{\ast b}$

• For each bootstrap sample, fit regression and estimate S.E. and C.I.

Wild Bootstrap

When variance of error $Var({ϵ}_i|X_i)$ depends on the value of $X_i$ ( so called heteroskedasticity) , residual bootstrap is unstable because the residual bootstrap will swap all the residuals regardless of the value of X. But wild bootstrap uses the residual of itself only.

• Generate IID random variables $V_1^b,...,V_n^b\sim N(0,1)$

• Generate new bootstrap sample: $X_i^{\ast b}=X_i,Y_i^{\ast b}={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i+V_i^b{\hat{e}}_i$

• For each bootstrap sample, fit regression and estimate S.E. and C.I.