NYU/CNS : Faculty : Core Faculty : Eero P. Simoncelli

My research addresses a variety of basic issues in the analysis and representation of visual images. The work is interdisciplinary, spanning computational neuroscience, image processing, and computer vision. Three broad goals provide the motivation for this work: 1) construction of mathematical theories for the representation of visual information, 2) development of functional models for biological visual processing, and 3) creation of novel algorithms for image processing and computer vision applications. Current major research topics include:

Statistical models of visual images. When coupled with a statistical characterization of their coefficients, wavelets can provide a "prior" model for natural images. We have used such prior models in a number of image processing applications (e.g., image compression, noise removal and image enhancement, analysis and synthesis of texture). Furthermore, the computations required to properly decompose signals generated according to these statistical models are remarkably similar to the divisive normalization computation used in neural models (see above). As such, these models of natural image statistics provide a theoretical justification for these neural models, and offer the opportunity to test directly (through physiological measurements) the ecological hypothesis that visual neural computations are optimally matched to the statistics of images.
Model of visual motion processing. This is a functional model, derived from fundamental properties of motion information encoded in visual signals. The computation is performed in two stages of identical architecture, corresponding to neurons in cortical areas V1 and MT. Each stage computes a weighted linear sum of inputs, followed by rectification and divisive normalization. The output of the model corresponds to the steady-state firing rates of a population of MT neurons, which form a distributed representation (population encoding) of image velocity for each local spatial region of the visual stimulus. The model accounts for a wide range of physiological data, and a related model is consistent with a surprising variety of human perceptual phenomena.
Functional characterization of neural response. The functional properties of sensory neurons have been traditionally summarized using "receptive fields." But these do not provide a complete description of the response properties unless one makes additional simplifying assumptions (e.g., linearity). Furthermore, as sensory neuroscience research has been extended to areas that are farther removed from the sensory input, it has become increasingly difficult to describe the receptive fields of neurons, because it is difficult to construct parametric stimuli that elicit responses. Recently my laboratory has been developing new forms of stimuli and data analysis techniques for probing and characterizing neurons, specifically techniques for identifying and estimating various forms of nonlinear response behavior, such as short-timescale gain adjustments or nonlinearities associated with spike generation. We are also using the statistical models described above to explore the generation of stochastic stimuli with "naturalistic" properties, which we hope will be more effective at eliciting neuronal responses. We've been working to develop new methodologies for estimating and testing models for the response of single neurons or small populations of neurons.
Multi-scale multi-orientation image representation. Starting with the work of my Master's Thesis in 1986, I've devoted a large proportion of my research efforts toward development of representations of this type, suitable for visual image processing. Recent work includes the development of the steerable pyramid representation that, unlike standard "wavelet" representations, has both translation- and rotation-invariance properties. This is essentially a tool for performing multi-scale differential analysis of imagery. Current work includes applications of these representations to problems of orientation analysis and rotation-invariant pattern matching, and texture and motion processing.

I started my higher education as a physics major at Harvard, went to Cambridge University on a Knox Fellowship to study Mathematics for a year and a half, and then returned to the States to pursue a doctorate in Electrical Engineering and Computer Science at MIT. I received my Ph.D. in 1993, and joined the faculty of the Computer and Information Science department at U Pennsylvania. I came to NYU in September of 1996, as part of the Sloan Center for Theoretical Visual Neuroscience. I received an NSF Faculty Early Career Development (CAREER) grant in September '96, for research and teaching in "Visual Information Processing", and a Sloan Research Fellowship in February of 1998. In August 2000, I became an Investigator of the Howard Hughes Medical Institute, under their new program in Computational Biology.

Representative Publications

Full publication list, w/ online reprints
List at Google Scholar, w/ related articles
Partial listing at PubMed

EP Simoncelli, EH Adelson, and DJ Heeger. Probability distributions of optical flow. In . Proc Conf on Computer Vision and Pattern Recognition, pp 310-315, Mauii, Hawaii, June 3-6 1991.

EP Simoncelli, WT Freeman, EH Adelson and DJ Heeger. Shiftable multi-scale transforms [or, what's wrong with orthonormal wavelets?]. IEEE Trans. Information Theory, 38(2):587-607, Mar 1992.

EP Simoncelli and DJ Heeger. A model of neural responses in visual area MT. Vision Research, 38(5):743-761, Mar 1998.

PR Schrater and EP Simoncelli. Local velocity representation: Evidence from motion adaptation. Vision Research , 38(24):3899-3912, Dec 1998.

EP Simoncelli. Bayesian multi-scale differential optic flow. Vol 2, ch 14 of Handbook on Computer Vision and Applications, eds. B Jähne, H Haussecker, and P Geissler. pp 397-422, Apr 1999. Academic Press, San Diego.

RW Buccigrossi and EP Simoncelli. Image compression via joint statistical characterization in the wavelet domain. IEEE Trans. Image Processing, 8(12):1688-1701, Dec 1999.

PR Schrater, DC Knill, and EP Simoncelli. Mechanisms of visual motion detection. Nature Neuroscience, 3(1):64-68, Jan 2000.

J Portilla and EP Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. Int'l Journal of Computer Vision, 40(1):49-71, Dec 2000.

O Schwartz and EP Simoncelli, Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8):819-825, Aug 2001.

Y Weiss, EP Simoncelli, and EH Adelson. Motion illusions as optimal percepts. Nature Neuroscience, 5(6):598-604, Jun 2002.

J Portilla, V Strela, M Wainwright and EP Simoncelli. Image denoising using a scale mixture of Gaussians in the wavelet domain. IEEE Trans. Image Processing, 12(11):1338-1351, Nov 2003.

EP Simoncelli. Local analysis of visual motion. Chapter 109 in The Visual Neurosciences, Eds. L M Chalupa and J S Werner, MIT Press, Nov 2003.

Z Wang, AC Bovik, HR Sheikh and EP Simoncelli. Image quality assessment: From error measurement to structural similarity. IEEE Trans. Image Processing, 13(4):496-508, Apr 2004.

EP Simoncelli, J Pillow, L Paninski and O Schwartz. Characterization of neural responses with stochastic stimuli. Chapter 23 in The Cognitive Neurosciences, 3rd edition, Ed: M Gazzaniga. MIT Press, Oct 2004.

L Paninski, J Pillow and EP Simoncelli. Maximum likelihood estimation of a stochastic integrate-and-fire neural encoding model. Neural Computation, 16(12):2533-2561, Dec 2004.

EP Simoncelli Statistical modeling of photographic images. Chapter 4.7 in Handbook of Image and Video Processing, 2nd edition, ed. Alan Bovik, Academic Press, 2005.

NC Rust, O Schwartz, JA Movshon, and EP Simoncelli. Spatiotemporal elements of Macaque V1 receptive fields. Neuron, 46(6):945-956, June 2005.

Z Wang and EP Simoncelli. An adaptive linear system framework for image distortion analysis. Proc Int'l Conf Image Processing, Genoa Italy, Sep 2005.

J Pillow, L Paninski, VJ Uzell, EP Simoncelli, and EJ Chichilnisky. Prediction and decoding of retinal responses with a probabilistic spiking model. J. Neuroscience, 25(47):11003-11013, Nov 2005.

J Malo, EP Simoncelli, I Epifanio and R Navarro. Non-linear image representation for efficient perceptual coding. IEEE Trans. Image Processing, 15(1):68-80, Jan 2006.

A Stocker and EP Simoncelli. Noise characteristics and prior expectations in human visual speed perception. Nature Neuroscience, 9(4):578--585, Apr 2006.

J Pillow and EP Simoncelli. Dimensionality reduction in neural models: An information-theoretic generalization of spike-triggered average and covariance analysis. Journal of Vision, 6(4):414-428, May 2006.

NC Rust, V Mante, EP Simoncelli, and JA Movshon. How MT cells analyze the motion of visual patterns. Nature Neuroscience, 9(11):1421-1431, Nov 2006.

S Lyu and E P Simoncelli. Statistically and perceptually motivated nonlinear image representation Proc. SPIE Conf. on Human Vision and Electronic Imaging XII, Jan 2007.

A Stocker and EP Simoncelli. A Bayesian model of conditioned perception. In Adv. Neural Information Processing Systems (NIPS*07), v20, May 2008.

S Lyu and E P Simoncelli. Modeling wavelet subbands of photographic images with fields of Gaussian scale mixtures. IEEE Trans. Patt. Analysis and Machine Intelligence, Online version published May 2008.

M Raphan and E P Simoncelli. Optimal denoising in redundant representations. IEEE Trans Image Processing, Aug 2008.

J W Pillow, J Shlens, L Paninski, A Sher, A M Litke, E J Chichilnisky, and E P Simoncelli. Spatio-temporal correlations and visual signaling in a complete neuronal population. Nature, Aug 2008.

Z Wang and E P Simoncelli. Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual discriminability, Journal of Vision, Sep 2008.

S Lyu and E P Simoncelli. Nonlinear extraction of 'Independent Components' of natural images using radial Gaussianization, Neural Computation, Jun 2009.

A Stocker and E P Simoncelli. Visual motion aftereffects arise from two isomorphic adaptation mechanisms Journal of Vision, 2009.

P Series, A A Stocker and E P Simoncelli. Is the homunculus 'aware' of sensory adaptation? Neural Computation, 2009.