Faculty of Arts & Science Site NYU Main Site NYU Main Site
NYU/CNS - Center for Neural Science New York University - Faculty of Arts & Sciences
FAS CNS Site
Core Faculty
Global Distinguished Professor
Associates
Affiliates
Research Scientists
Visiting Scholars
Postdoctoral
Predoctoral


Search CNS
  • CNS Personnel Directory (Local Only)

    • Eero P. Simoncelli

    Computational Vision
    		 Home page  |  Lab page  |  FlatIron Institute page

    Our sensory systems provide us with a remarkably reliable interpretation of the world, allowing us to make predictions and perform difficult tasks with surprising accuracy. How do these capabilities arise from the underlying neural circuitry? Specifically, how do populations of neurons encode sensory information, and how do subsequent populations extract that information for recognition, decisions, and action? And from a more theoretical perspective, why do sensory systems use these particular representations, and how can we use these principles to design better man-made systems for processing sensory signals? Broadly speaking, my research aims to answer these questions, through a combination of computational theory and modeling, coupled with perceptual and physiological experiments. These endeavors can be categorized into three general classes.

  • Optimal Encoding of Visual Information. Image statistics and cortical normalization models It has long been assumed that visual systems are adapted, at evolutionary, developmental, and behavioral timescales, to the images to which they are exposed. Since not all images are equally likely, it is natural to assume that the system use its limited resources to process best those images that occur most frequently. Thus, it is the statistical properties of the environment that are relevant for sensory processing. Such concepts are fundamental in engineering disciplines -- compression, transmission, and enhancement of images all rely heavily on statistical models. Since the mid 1990's we've developed successively more powerful models describing the statistical properties of local regions of natural images [ref1, ref2, ref3], demonstrated the power of these models by using them to develop state-of-the-art solutions to classical engineering problems of compression and noise removal, and using them in parallel to understand the structure and function of both visual and auditory neurons. These same ideas can be used to construct new nonlinear forms of image representation [ref] that provide a framework for assessing perceptual distortion [ref1, ref2, ref3]. In a related line of research, we've explored image representations that offer various forms of "invariance". This includes the development of the steerable pyramid image representation that serves as a substrate for most of our image processing and computer vision applications, as well as modeling the receptive fields of populations of neurons in primary visual cortex. Recent work includes the development of new forms of signal-adaptive representations [ref1, ref2, ref3].
  • Experimental Characterization of Neural and Perceptual Responses. Our models for sensory representations serve as precise instantiations of scientific hypotheses, and must therefore be tested and refined through comparison to experimental measurements. A component of our work is aimed at developing new experimental paradigms, including novel stimuli and analysis methods, for such experiments. In the retina we find that a "general linear model" (GLM), in which spiking responses arise from the superposition of a filtered stimulus signal, a feedback signal (embodying refractoriness and other forms of suppression excitation derived from the spike history), and a lateral connectivity signal (embodying influences from the spiking activity of other cells) provides a remarkably precise account of spike timing in populations of ganglion cells. In primary visual cortex (area V1), we've developed and fit a model that can capture the stimulus selectivity and gain control properties of a wide range of cells. In extrastriate cortex, we've developed and refined a model for motion representation in the middle temporal (MT) dorsal area. More recently, we've developed targeted stochastic motion stimuli that allow us to characterize the specific properties of individual MT neurons in terms of their V1 afferents. And by examining human detection performance for such stimuli, we have produced strong evidence for the existence of such mechanisms in the human visual system. We've also developed a model for the representation of visual texture, and used it to synthesize texture images that humans perceive as similar (we've developed analogous models for auditory textures). By coupling this model with known receptive field properties of neurons in the ventral stream (specifically, the growth of receptive field size with eccentricity), we've been able to generate new forms of stimuli that exhibit severe peripheral distortion (scrambling of visual patterns, and a complete loss of recognizability) but are indistinguishable from intact photographs ref. We've used perceptual experiments to determine the sizes of neural receptive fields underlying these ambiguities, which allows us to identify the locus of this representation as area V2. Physiological experiments are currently underway (in collaboration with the Movshon lab) to further elucidate these neural mechanisms.
  • Optimal Decoding & Perception Our everyday experience deludes us into believing that perception is a direct reflection of the physical world around us. But scientists have recognized for centuries that it is more akin to a process of inference, in which incoming measurements are fused with internal expectations. In the 20th century, this concept was formalized in theories of Bayesian statistical inference, and since the early 1990s, I've used this framework to understanding the means by which percepts arise from neural responses. An interesting example arises in the perception of retinal motion. If one assumes that the light intensity pattern falling on a local patch of retina is undergoing translational motion, that the neural representation of this information is noisy, and that in the absence of visual information, the distribution of retinal velocities that are typically encountered is broad but centered at zero (no motion), one can derive an optimal estimator for image velocity [ref1, ref2]. The resulting estimates are strongly biased toward slower speeds when the incoming stimulus is weakened (e.g., at low contrast). This behavior is also seen in humans, and we've used perceptual measurements to determine the internal preferences of human observers. We've obtained analogous results for human perception of local orientation, where observer preferences for horizontal and vertical orientations are well-matched to their prevalence in the natural world. The inferential computations required for these percepts are compatible with the simple neural models described above, and our current work (both theoretical and experimental) aims to elucidate the means by which prior preferences can be learned and embedded in neural populations.

    • Biography: I started my higher education as a physics major at Harvard, went to Cambridge University on a Knox Fellowship to study Mathematics for a year and a half, and then returned to the States to pursue a doctorate in Electrical Engineering and Computer Science at MIT. I received my Ph.D. in 1993, and joined the faculty of the Computer and Information Science department at U Pennsylvania. I came to NYU in September of 1996, as part of the Sloan Center for Theoretical Visual Neuroscience. I received an NSF Faculty Early Career Development (CAREER) grant in September '96, for research and teaching in "Visual Information Processing", and a Sloan Research Fellowship in February of 1998. In August 2000, I became an Investigator of the Howard Hughes Medical Institute, under their new program in Computational Biology. In Fall of 2020, I resigned from HHMI to become the Director of the Center for Computational Neuroscience at the FlatIron Institute of the Simons Foundation (while remaining a faculty member at NYU).

    Selected Representative Publications
    (chronological)

    		 Complete LCV listing, with reprints
    Google Scholar
    Semantic Scholar
    OpenReview
    ResearchGate
    Loop
    ORCID
    PubMed

    EP Simoncelli, WT Freeman, EH Adelson and DJ Heeger. Shiftable multi-scale transforms [or, what's wrong with orthonormal wavelets?]. IEEE Trans. Information Theory, 38(2):587-607, Mar 1992.

    EP Simoncelli. Statistical models for images: Compression, restoration and synthesis. Proc. 31st Asilomar Conf on Signals, Systems and Computers, vol.1, pp. 673--678, Nov 1997.

    EP Simoncelli and DJ Heeger. A model of neural responses in visual area MT. Vision Research, 38(5):743-761, Mar 1998.

    EP Simoncelli. Bayesian multi-scale differential optic flow. Vol 2, ch 14 of Handbook on Computer Vision and Applications, eds. B Jähne, H Haussecker, and P Geissler. pp 397-422, Apr 1999. Academic Press, San Diego.

    PR Schrater, DC Knill, and EP Simoncelli. Mechanisms of visual motion detection. Nature Neuroscience, 3(1):64-68, Jan 2000.

    J Portilla and EP Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. Int'l Journal of Computer Vision, 40(1):49-71, Dec 2000.

    EP Simoncelli and B Olshausen. Natural image statistics and neural representation. Annual Review of Neuroscience, 24:1193-1216, May 2001.

    O Schwartz and EP Simoncelli, Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8):819-825, Aug 2001.

    MJ Wainwright, EP Simoncelli and AS Willsky, Random cascades on wavelet trees and their use in analyzing and modeling natural images. Applied and Computational Harmonic Analysis, 2001.

    Y Weiss, EP Simoncelli, and EH Adelson. Motion illusions as optimal percepts. Nature Neuroscience, 5(6):598-604, Jun 2002.

    J Portilla, V Strela, M Wainwright and EP Simoncelli. Image denoising using a scale mixture of Gaussians in the wavelet domain. IEEE Trans. Image Processing, 12(11):1338-1351, Nov 2003.

    Z Wang, AC Bovik, HR Sheikh and EP Simoncelli. Image quality assessment: From error measurement to structural similarity. IEEE Trans. Image Processing, 13(4):496-508, Apr 2004.

    EP Simoncelli, J Pillow, L Paninski and O Schwartz. Characterization of neural responses with stochastic stimuli. Chapter 23 in The Cognitive Neurosciences, 3rd edition, Ed: M Gazzaniga. MIT Press, Oct 2004.

    NC Rust, O Schwartz, JA Movshon, and EP Simoncelli. Spatiotemporal elements of Macaque V1 receptive fields. Neuron, 46(6):945-956, June 2005.

    A Stocker and EP Simoncelli. Sensory adaptation within a Bayesian framework for perception. Adv. Neural Information Processing Systems (NIPS*05), v18, May 2006.

    A Stocker and EP Simoncelli. Noise characteristics and prior expectations in human visual speed perception. Nature Neuroscience, 9(4):578--585, Apr 2006.

    J Pillow and EP Simoncelli. Dimensionality reduction in neural models: An information-theoretic generalization of spike-triggered average and covariance analysis. Journal of Vision, 6(4):414-428, May 2006.

    S Lyu and E P Simoncelli. Modeling multiscale subbands of photographic images with fields of Gaussian scale mixtures. IEEE Trans. Patt. Analysis and Machine Intelligence, 31(4):693-706, Apr 2009.

    J W Pillow, J Shlens, L Paninski, A Sher, A M Litke, E J Chichilnisky, and E P Simoncelli. Spatio-temporal correlations and visual signaling in a complete neuronal population. Nature, 454(7206):995-999, Aug 2008.

    S Lyu and E P Simoncelli. Nonlinear extraction of 'Independent Components' of natural images using radial Gaussianization, Neural Computation, 21(6):1485-1519, Jun 2009.

    A Stocker and E P Simoncelli. Visual motion aftereffects arise from two isomorphic adaptation mechanisms, Journal of Vision, 9(9):1-14, 2009.

    E P Simoncelli. Optimal estimation in sensory systems , Chapter 36 in "The Cognitive Neurosciences, IV", Ed. M Gazzaniga, Oct 2009.

    P Series, A A Stocker and E P Simoncelli. Is the homunculus 'aware' of sensory adaptation? Neural Computation, 21(12):3271-3304, Dec 2009.

    M Raphan and E P Simoncelli. Least squares estimation without priors or supervision. Neural Computation, 23(2):374-420, Nov 2010.

    Y Karklin and E P Simoncelli. Efficient coding of natural images with a population of noisy linear-nonlinear neurons, Adv. Neural Information Processing Systems (NIPS*10), v23, May 2011.

    JH Hedges, AA Stocker, and EP Simoncelli. Optimal inference explains the perceptual coherence of visual motion stimuli, Journal of Vision, 11(6), May 2011.

    AR Girshick, MS Landy, and EP Simoncelli. Cardinal rules: Visual orientation perception reflects knowledge of environmental statistics, Nature Neuroscience, 14(7):926-932, Jul 2011.

    J Freeman and E P Simoncelli. Metamers of the ventral stream, Nature Neuroscience, 14(9):1195-1201, Sep 2011.

    C Ekanadham, D Tranchina, and EP Simoncelli. Recovery of sparse translation-invariant signals with continuous basis pursuit, IEEE Trans Signal Processing, 59(10):4735-4744, Oct 2011.

    E Doi, J Gauthier, G Field, J Shlens, A Sher, M Greschner, T Machado, L Jepson, K Mathieson, D Gunning, A Litke, L Paninski, EJ Chichilnisky and E P Simoncelli. Efficient coding of spatial information in the primate retina, J. Neuroscience, Nov 2012.

    J H McDermott, M Schemitsch and E P Simoncelli. Summary statistics in auditory perception, Nature Neuroscience, Apr 2013.

    J Pillow, J Shlens, EJ Chichilnisky and E P Simoncelli. A model-based spike sorting algorithm for reducing correlation artifacts in multi-neuron recordings, PLoS One, May 2013.

    J Freeman, C Ziemba, D J Heeger, E P Simoncelli and J A Movshon. A functional and perceptual signature of the second visual area in primates, Nature Neuroscience, Jul 2013.

    C Ekanadham, D Tranchina and E P Simoncelli. A unified framework and method for automatic neural spike identification, J. Neuroscience Methods 2013.

    R L Goris, J A Movshon and E P Simoncelli. Partitioning neuronal variability, Nature Neuroscience, 2014.

    D Ganguli and E P Simoncelli. Efficient sensory coding and Bayesian decoding with neural populations, Neural Computation, 2014.

    O Henaff, N Rabinowitz, J Ballé and E P Simoncelli. The local low-dimensionality of natural images, Intl Conf on Learning Representations (ICLR), May 2015.

    J Freeman, G D Field, P H Li, M Greschner, D H Gunning, K Mathieson, A Sher, A M Litke, L Paninski, E P Simoncelli and E J Chichilnisky. Mapping nonlinear receptive field subunits in primate retina at single cone resolution, eLife, 2015.

    B Vintch, J A Movshon and E P Simoncelli. A convolutional subunit model for neuronal responses in macaque V1, J Neurosci, 2015.

    R L Goris, E P Simoncelli and J A Movshon. Origin and function of tuning diversity in macaque visual cortex, Neuron, 2015.

    E Ganmor, M S Landy and E P Simoncelli. Near-optimal integration of orientation information across saccades, Journal of Vision, 2015.

    N C Rabinowitz, R L Goris, M Cohen and E P Simoncelli. Attention stabilizes the shared gain of V4 populations, eLife, 2015.

    D Ganguli and E P Simoncelli. Neural and perceptual signatures of efficient sensory coding, arXiv.org 1603.00058, Feb 2016.

    J Ballé, V Laparra and E P Simoncelli. Density modeling of images using a generalized normalization transformation, Int'l Conf on Learning Representations (ICLR), 2016.

    CM Ziemba, J Freeman, JA Movshon and EP Simoncelli. Selectivity and tolerance for visual texture in macaque V2, Proc. National Academy of Sciences, 2016.

    M Pagan, E P Simoncelli and N C Rust. Neural quadratic discriminant analysis: Nonlinear decoding with V1-like computation, Neural Computation, 2016.

    J Ballé, V Laparra and E P Simoncelli. End-to-end optimized image compression, Int'l Conf on Learning Representations (ICLR), 2017.

    V Laparra, A Berardino, J Ballé and E P Simoncelli. Perceptually optimized image rendering, J. Optical Society of America A, 2017.

    A Berardino, V Laparra, J Ballé and E P Simoncelli. Eigen-distortions of hierarchical representations, Adv. Neural Information Processing Systems (NIPS*17), Dec 2017.

    OJ Henaff, RLT Goris and EP Simoncelli. Perceptual straightening of natural videos, Nature Neuroscience, 2019.

    C Haimerl, C Savin and E P Simoncelli. Flexible information routing in neural populations through stochastic comodulation, Adv. Neural Information Processing Systems (NeurIPS), 2019.

    S Mohan*, Z Kadkhodaie*, E P Simoncelli and C Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks, Int'l. Conf. on Learning Representations (ICLR), Apr 2020.

    N Parthasarathy and E P Simoncelli. Self-supervised learning of a biologically-inspired visual texture model, arXiv.org e-prints 2006.16976, Jun 2020.

    Z Kadkhodaie and E P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser, arXiv.org 2007.13640, Jul 2020.

    C Bredenberg, E P Simoncelli and C Savin. Learning efficient task-dependent representations with synaptic plasticity, Adv. Neural Information Processing Systems (NeurIPS*20), Dec 2020.

    K Ding, K Ma, S Wang and E P Simoncelli. Image quality assessment: Unifying structure and texture similarity, IEEE Trans. Patt. Analysis and Machine Intelligence, Dec 2020.

    CM Ziemba and EP Simoncelli. Opposing effects of selectivity and invariance in peripheral vision, Nature Communications, 2021.

    JY Zhou, LR Duong and EP Simoncelli. A common framework for discriminability and perceived intensity of sensory stimuli, bioRxiv 2022.04.30.490146, May 2022.

    E G Wu, N Brackbill, A Sher, A M Litke, E P Simoncelli and E J Chichilnisky. Maximum a posteriori natural scenes reconstruction from retinal ganglion cells with deep denoiser priors Adv. Neural Information Processing Systems (NeurIPS*22), Dec 2022.

    Z Kadkhodaie, F Guth, S Mallat and E P Simoncelli. Learning multi-scale local conditional probability models of images, Int'l Conf on Learning Representations (ICLR), May 2023.

    C Haimerl, D A Ruff, M R Cohen, C Savin and E P Simoncelli. Targeted V1 comodulation supports task-adaptive sensory decisions, Nature Communications, Nov 2023.

    TE Yerxa, Y Kuang, EP Simoncelli and SY Chung. Efficient coding of natural images using maximum manifold capacity representations, Adv. Neural Information Processing Systems (NeurIPS), Dec 2023.

    L R Duong, E P Simoncelli, D B Chklovskii and D Lipshutz. Adaptive whitening with fast gain modulation and slow synaptic plasticity, Adv. Neural Information Processing Systems (NeurIPS), Dec 2023.

    P-E Fiquet and E P Simoncelli. Neural representations for predictive processing of dynamic visual signals, Adv. Neural Information Processing Systems (NeurIPS), Dec 2023.

    N Parthasarathy, O J Hénaff and E P Simoncelli. Layerwise complexity-matched learning yields an improved model of cortical area V2, Trans. Machine Learning Research, Jun 2024. Recipient, Featured Article certification.

    Z Kadkhodaie, F Guth, E P Simoncelli and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representation, Int'l Conf on Learning Representations (ICLR), May 2024. Recipient, Outstanding Paper award.

    D Lipshutz and E P Simoncelli. Shaping the distribution of neural responses with interneurons in a recurrent circuit model, Adv. Neural Information Processing (NeurIPS), Dec 2024.

    T Yerxa, J Feather, E P Simoncelli and SY Chung. Contrastive-equivariant self-supervised learning improves alignment with primate visual area IT, Adv. Neural Information Processing (NeurIPS), Dec 2024.

    X Niu, C Savin and E P Simoncelli. Learning predictable and robust neural representations by straightening image sequences, Adv. Neural Information Processing (NeurIPS), Dec 2024.

    J Feather*, D Lipshutz*, S E Harvey, A H Williams and E P Simoncelli. Discriminating image representations with principal distortions, Int'l Conf on Learning Representations (ICLR), May 2025.

    		 

    Back to top

    | | | Accessibility
    All contents © New York University. All rights reserved.
    Revised: 30-Jan-2025 7:38PM