Depth Changes Scaling, Not Just Capacity

Chapter 6, Deep Feedforward Networks, separates three questions that are easy to blur: what a network can represent, how efficiently it can represent it, and whether optimization can find the parameters. Architecture is not just a tuning dimension. It is a structural assumption about the function class we are willing to search.

Existence Does Not Imply Learnability

The common shorthand for the Universal Approximation Theorem is that neural networks can approximate any function. Chapter 6 makes the important constraint explicit: this is an existence result about representational capacity, not a statement about efficiency or trainability.

The useful split has three parts. First, representational capacity: what functions are expressible in principle. Second, parameter efficiency: how many units are required to represent them. Third, learnability: whether optimization can reliably discover those parameters from data. The theorem addresses only the first.

The split is useful when comparing shallow and deep networks. A single hidden layer network can approximate complex structured functions, but the width required may scale exponentially for certain compositional forms. Depth can reduce parameter count by reusing intermediate computations. In that sense, depth changes scaling behavior, not just capacity.

The most useful refinement was separating topology from optimization. The architecture defines the hypothesis class; gradient descent explores it imperfectly. The fact that a function is representable says nothing about whether it is practically learnable. That separation gives me a cleaner way to evaluate model design choices.

Linear depth imposes a rank constraint

One useful structural point is what happens when we stack linear layers without nonlinearities between them. Two consecutive linear transformations collapse into a single linear transformation. Functionally, nothing changes, but the parameterization does.

If a weight matrix $W \in \mathbb{R}^{m \times n}$W∈Rm×n is factored as $W = AB$W=AB with $A \in \mathbb{R}^{m \times r}$A∈Rm×r and $B \in \mathbb{R}^{r \times n}$B∈Rr×n, we have expressed the same linear map with a rank constraint and potentially far fewer parameters when $r \ll \min(m, n)$r≪min(m,n). In other words, a linear path with a narrow intermediate layer induces a low-rank factorization.

The same structure shows up in LoRA-style adaptation for large language models. Inserting a bottleneck linear path imposes a low-rank constraint on the effective weight update. The connection is basic linear algebra: the architecture changes the parameterization of the update.

This reframed low-rank adaptation for me. The architecture itself encodes a rank prior.

Softplus vs ReLU

The comparison between softplus and ReLU corrected an intuition I had inherited from smooth optimization: smoother functions should be easier to train. Softplus is differentiable everywhere with nonzero gradient, while ReLU is nondifferentiable at zero and flat for negative inputs. By a classical smoothness criterion, softplus seems preferable.

Empirically, ReLU often performs better. Chapter 6 provides a structural explanation. ReLU induces sparsity through hard gating, effectively selecting a subnetwork conditioned on the input. The resulting function is piecewise linear: globally nonlinear, but locally linear within each region. The training behavior is not governed by smoothness alone.

Smoothness alone does not determine learnability. Geometry, sparsity, and gating behavior shape gradient flow and effective model complexity. That makes me more cautious about mapping classical smooth optimization intuitions directly onto deep networks.

Architecture defines the search space

Chapter 6 is less a tuning guide than a sharper vocabulary for model structure. I now think less in terms of "can this network approximate the function?" and more in terms of parameter efficiency, structural priors, and the separation between representation and optimization. Neural networks feel less like universal approximators and more like structured hypothesis classes with specific geometric and algebraic constraints.