When I worked through Chapter 7 of Deep Learning, the bagging interpretation of dropout was the most useful part. The shortcut explanation treats dropout as random noise that helps regularization. Chapter 7 gives a more precise model: dropout works like a relatively inexpensive form of model averaging, close in spirit to bagging.
Dropout approximates model averaging
Bagging reduces variance by averaging predictions from models trained on resampled data. Training and serving separate networks is expensive. Dropout instead samples thinned networks that share parameters, then uses the full network to approximate their combined predictions. These are not independently trained models, and the approximation is not generally an exact average.
The original dropout paper scales weights at inference. Inverted dropout, used by PyTorch, scales retained activations during training instead, so no dropout scaling is needed at inference.
Changing masks discourages co-adaptation
The bagging interpretation also makes another part of dropout easier to think about: the apparent paradox that it can widen the set of subnetworks seen during training while still regularizing. Each binary mask defines a different thinned network, but those networks share weights. At the same time, any one unit cannot rely on a specific partner always being present, so representations are pushed to be useful across many subnet configurations.
That pressure reduces fragile co-adaptation. Features that only work in one narrow pathway get penalized indirectly, while more robust features survive across many sampled masks. I like this framing because it explains dropout through the training setup itself, not just through the shortcut phrase "add noise."
Get new posts by email
Occasional updates when I publish something new.
Follow.it sends the emails and includes an unsubscribe link.