Gradient Boosting
A random forest builds all its trees at once and averages them. Gradient boosting is sneakier and often stronger: it builds trees one after another, in a chain, where every new tree exists only to fix the mistakes the chain has made so far. It is the model that wins real prediction competitions. Watch it correct itself, round by round.
The one idea: keep fixing the leftovers
Start with a lazy guess: predict the plain average for every day. Now look at what is left over, the errors. Train one small tree just to predict those errors, add a careful fraction of it, and your guess improves a little. Look at the new, smaller leftovers. Train another tree on those. Repeat. Each tree stands on the shoulders of the last.
The residual is what your current prediction still gets wrong, the gap between the real value and your guess. Boosting never predicts the price directly after the first step. Instead each new tree predicts the residuals, chipping the error down toward zero.
Run the chain yourself
The gold line is the model's current prediction; it starts flat at the average. The coral sticks are the leftover errors. Press Add a correcting tree and watch the gold line bend toward the dots as the coral sticks shrink.
How to play
- Press Add a correcting tree. Each one targets the current coral errors.
- Watch the gold prediction hug the teal dots more tightly every round.
- Read the training error fall. It keeps dropping, which is exactly why you must be careful not to overdo it.
Keep adding trees and the training error marches toward zero, but past a point the model is just memorising the exact days, including their noise. On new data that hurts. Real practitioners use a small learning rate and stop early, trading a perfect fit on the past for a better guess on the future.
What it is great at, and where it slips
Squeezing the most accuracy out of a table of features. Versions like XGBoost win competition after competition on data like ours.
It is left to run too long or tuned carelessly. It will happily overfit, and like all trees it cannot extrapolate past seen levels.
The boosting mindset is useful even by hand: make a rough guess for next month's bill (say, last month's amount), then ask "what did I forget?" More hot days? A new appliance? Correct your guess for each. That is boosting in plain life, and you will practise it in the bonus light-bill challenge.
- Gradient boosting builds trees in a chain, each fixing the last one's leftover errors.
- Every tree after the first predicts residuals, driving the error down step by step.
- It is extremely accurate but overfits if you run too many rounds, so use a small learning rate and stop early.
Next: LSTM, a network that carries a memory across a sequence.