Sensory neurons encode stimulus information with sequences of action potentials that are inherently probabilistic: repeated measurements under identical stimulus conditions yield different spike trains. This variability is usually treated as noise arising from stochasticity in neurons and neural circuits. However, in many experimental situations, a portion arises from fluctuations in neural excitability caused by uncontrolled modulatory sources such as adaptation, attention, arousal, and alertness. To separate these contributions from response noise, we make use of a model in which spikes are generated by a Poisson process whose rate is the product of experimentally controlled stimulus-dependent excitation and randomly fluctuating excitability. We fit this model to responses of visual neurons of anesthetized macaques elicited by sets of drifting sinusoidal gratings containing both near-optimal and suboptimal stimuli. The model provides an excellent account of individual neuronal responses in LGN and cortical areas V1, V2, and MT. The fitted model separates the effects of spiking noise from uncontrolled fluctuations in excitability, revealing that the variance of excitability fluctuations systematically increases in strength along the visual hierarchy, and is the dominant source of response variance in cortex, but not in thalamus. Using data from multi-electrode array recordings in V1, we also explored the spatial and temporal structure of these excitability fluctuations. We estimated excitability on a trial-by-trial basis and found fluctuations to be correlated over relatively long timescales (1-15 min), and between proximal neurons (within 4 mm of cortex). We conclude that the response variability of sensory neurons originates in large part from slow fluctuations in excitability that are shared across neurons and that increase in strength along the visual pathway.