Bootstrap Aggregating

Definition

Regression Setting

Given a training set $Z=\{(x_i,y_i)|1=1,\dots ,n\}$, draw a bootstrap sample $\{Z^{\ast b}|b=1,\dots ,B\}$ from training data. Bagging estimator of $f$ at $x$ is defined as ${\hat{f}}_{\text{bag}}(x)=\frac{1}{B}\sum _{b=1}^B{\hat{f}}^{\ast b}(x)$ where ${\hat{f}}^{\ast b}(x)$ is the prediction at $x$ based on a new training set $Z^{\ast b}$

Classification Setting

Given a training set $Z=\{(x_i,y_i)|1=1,\dots ,n\}$, draw a bootstrap sample $\{Z^{\ast b}|b=1,\dots ,B\}$ from training data. Bagging estimator of $f$ at $x$ is defined as ${\hat{f}}_{\text{bag}}(x)=\underset{1\le k\le K}{\mathrm{argmax}}\#\{b|{\hat{f}}^{\ast b}(x)=k\}$ where ${\hat{f}}^{\ast b}(x)$ is the prediction at $x$ based on a new training set $Z^{\ast b}$

Facts

Bagging in a classification setting with zero-one loss often fails. Bagging a good classifier can make it better, but bagging a bad classifier can make it worse.

Bagging reduces the variance of an estimated prediction function. It seems to work especially well for high-variance, low-bias procedures, such as trees.