Our model is derived from a novel objective, which reformulates whitening of neural responses in terms of the variances of a fixed overcomplete projection of the neural responses. Optimization of this objective yields an online algorithm that maps onto a recurrent neural network comprised of primary neurons, gain-modulating local interneurons and fixed synaptic weights. Each interneuron receives weighted inputs from the primary neurons, adjusts its input-output gain based on the variance of its input and feeds back onto the primary neurons, whitening their outputs. Our framework can be generalized to handle biophysical constraints that improve robustness of the network to ill-conditioned inputs, and we demonstrate its use in whitening local image patches using convolutional weights. We also consider a multi-timescale extension of our circuit model in which the synapses learn on a slow timescale. In this case, gains rapidly adapt to changing sensory statistics whereas synapses slowly adjust to learn invariant features of the sensory statistics.