Gradient Boosting

Definition

The gradient boosting is a type of Forward Stagewise Additive Modeling that uses gradients to calculate the residuals.

Algorithm

1. Initialize the model $f_0(x)=\underset{\mathbf{\gamma }}{\mathrm{argmin}}\sum _{i=1}^{n}L(y_i,\mathbf{\gamma })$
2. Repeat for $m=1,2,\dots ,M$:
   1. Calculate the residuals for each data point $r_{im}=-{\left[\frac{\partial L(y_i,f(x_i))}{\partial f(x_i)}\right]}_{f=f_{m-1}}$ where $L$ is a Loss Function
   2. Fit weak learners to the residuals $h_m(x)=\underset{h}{\mathrm{argmin}}\sum _{i=1}^n(r_{im}-h(x_i){)}^2$
   3. Calculate a learning rate ${\gamma }_m=\underset{\gamma }{\mathrm{argmin}}\sum _{i=1}^{n}L(y_i,f_{m-1}(x_i)+\gamma h_m(x_i))$
   4. Update model using the fitted weak learners $f_m(x)=f_{m-1}(x)+\gamma h_m(x)$
3. Output the final model $\hat{f}(x)=f_0(x)+\sum _{i=1}^m{\gamma }_m{h}_m(x)$