Optimization theory is the study of finding minima, and the convex versus non-convex split is the whole story in miniature. A convex problem has one basin; every local minimum is global, and the theory hands down convergence guarantees like verdicts. A non-convex problem, where every deep loss landscape lives, offers no such promise; gradient descent there is not a proof but a heuristic. That contrast, guarantee versus gamble on the same hillside, is what separates the two regimes.
Saddle points, not local minima, are the primary cause of slowdowns in high-dimensional training. A critical point where is a local minimum only when curvature is positive in every direction, but a saddle if any direction curves downward. In a million-dimensional space, having all directions curve the same way is vanishingly rare, so almost every flat region is a saddle, and training crawls not from being trapped but from gradients shrinking to near-zero on every side. Noise eventually finds the one direction still curving down.
Overparameterization smooths non-convex landscapes because extra parameters add directions to move in. When a network has far more weights than training points, the equations asking the network to fit the data perfectly admit a huge solution space, a manifold rather than a point. Interpolation becomes easy, and the landscape of interpolating solutions grows connected, so minima that would have been isolated traps in a narrow network merge into one wide valley. There is always slack left over: a direction that changes the weights while leaving predictions untouched. The optimization problem stays non-convex, but the valley floor stops being hostile.
The neural tangent kernel turns a deep net into linear regression by freezing its features at initialization. Defined as the inner product of the output gradients at two inputs, it stops moving in the infinite-width limit because the weights barely change. Training becomes gradient descent in function space under a fixed kernel: predictions drift linearly toward the targets, and because the kernel is positive semi-definite, the dynamics are convex with convergence guaranteed. The fixity is also the cost. No feature is ever learned, which is exactly why real finite-width nets outperform their NTK cousins.
Among the many interpolating solutions, implicit bias is the mechanism by which gradient descent picks just one. When the loss reaches zero on an overparameterized model, the solutions form an entire manifold, yet plain gradient descent converges to a specific point on it: the maximum-margin solution in simple linear models, and in deeper ones an interpolator shaped by the architecture and the initialization. Change the optimizer or the step size, and the choice changes too. Where convex theory once promised a unique answer, the non-convex world substitutes the algorithm’s own bias; the optimizer itself becomes the regularizer.