KKT Points and Ridgeless Interpolation in Symmetric ReLU Networks

KKT Points and Ridgeless Interpolation in Symmetric ReLU Networks

Theoretical Machine Learning

15th October 2026

Dominik Vuko

We consider two-layer ReLU networks with two symmetric neuron pairs and a skip connec- tion, constrained to interpolate four symmetric data points. We study the KKT points of the squared-norm minimisation problem, where the biases and skip connection are unpenalised. By constructing feasible descent directions, we show that every KKT point represents the unique ridgeless interpolant. We further establish that KKT points have balanced parame- ters and no cancellation between coincident slope jumps. Combining these results with the relationship between the squared norm and total variation, we prove that every feasible KKT point is a global minimiser of the squared-norm interpolation problem.

ReLU Neural Networks, KKT Conditions, Ridgeless Interpolation, Total Variation, Mathematical Optimisation, Minimum-Norm Interpolation