These effects can be captured by a modulated Poisson model, whose rate is the product of a stimulus-driven response function and an unknown modulatory signal (Goris, Movshon, Simoncelli, 2013). Here, we extend this model, by including modulatory elements that are known (specifically, spike-history dependence, as in previous GLM models, Pillow et al, 2008), and by constraining the remaining latent modulatory signals to be smooth in time. We fit the entire model, including hyperparameters, via evidence optimization (Park & Pillow, 2011), to the responses of ferret auditory midbrain and cortical neurons to complex sounds. Integrating out the latent modulators yields more readily-interpretable receptive field estimates than a standard Poisson model. Conversely, integrating out the stimulus dependence yields estimates of the slowly-varying latent modulators. For example, when applied to array recordings of macaque V1, we find complex spatial patterns of correlation amongst the latent modulators, including clusters of co-modulated units. In sum, use of the modulated Poisson model improves inference, and enables the study of signals underlying non-stationarities in neural responses.