We develop a methodology for estimating the curvature of internal trajectories from human perceptual judgments. We use this method to test three distinct predictions: natural sequences that are highly curved in the space of pixel intensities should be substantially straighter perceptually; in contrast, artificial sequences that are straight in the intensity domain should be more curved perceptually; finally, naturalistic sequences that are straight in the intensity domain should be relatively less curved. Perceptual data validate all three predictions, providing evidence that the visual system selectively straightens the temporal trajectories of natural image sequences.

If the visual hierarchy has learned to straighten the trajectories of natural videos, we would expect individual visual areas to contribute to perceptual straightening. We test this hypothesis by applying our curvature estimation methodology to population recordings from primary visual cortex, and find that the curvature of these neural trajectories is well predicted by our perceptual results. Finally, our hypothesis provides us with a framework for testing the metric properties of machine representations and their relationship to human vision. Together, these results point to the metric properties of natural videos as an effective way of identifying behaviorally relevant computation along the visual hierarchy.