This dissertation investigates two fundamental aspects of neural population coding: adaptive coding efficiency and stochastic representational geometry. We introduce a theory for adaptive statistical whitening revolving around a gain control mechanism, based on a novel overcomplete matrix factorization of the whitening transform. From this theory, we derive an online whitening algorithm that maps directly onto a recurrent neural network with primary neurons and an over-complete, auxiliary set of gain-modulating interneurons. Further elaborating on this framework, we integrate adaptive gain control with existing theories of adaptive whitening into a single unified adaptation objective using synaptic plasticity in a multi-timescale mechanistic model. This model adapts to changing sensory statistics by modifying gains and synapses at varying rates, resulting in improved adaptive whitening responses that is robust to non-stationary environments. Leveraging V1 population adaptation data, we demonstrate that propagation of single neuron gain changes through recurrent network structures is sufficient to explain the entire set of observed adaptation effects. Finally, we shift our focus to stochastic representational geometry, and introduce a family of distance metrics for comparing geometry between stochastic neural networks. These metrics are based on concepts from optimal transport theory and provide unique insights into the representations of noisy artificial and biological neural networks. Taken together, this thesis advances our understanding of neural population coding by examining the adaptive coding efficiency and the stochastic geometry of neural representations, with possible implications to the fields of neuroscience and machine learning.