Much of behavior relies on making predictions about future outcomes given recent observations. Yet visual input evolves according to complex, nonlinear dynamics that are difficult to extrapolate. We hypothesize that the brain transforms the incoming stream of images so as to make them more predictable. Specifically, we propose that internal representations are structured so as to straighten the trajectories of natural videos, thereby enabling prediction through linear extrapolation. In contrast, image sequences that are unlikely to occur in the real world need not be straightened and will most likely be distorted by the visual system. To test this "temporal straightening" hypothesis, we developed a novel procedure for estimating the curvature of the human perceptual representation of a sequence of images. Specifically, we first measured the discriminability of pairs of briefly presented video frames. Next, we formulated an observer model in which each frame is represented as a Gaussian distribution in a fixed-dimensional perceptual space. The likelihood of a trajectory can be evaluated by computing the probability that the measured human responses would have arisen from the overlap of the corresponding distributions. Finally, we derived a data-efficient and largely unbiased estimator of perceptual curvature by searching for the curvature that best describes all plausible perceptual trajectories. By comparing this value to the curvature calculated from the pixel intensities of the image sequence, we tested three distinct predictions of our hypothesis. First, natural videos that are curved in the intensity domain should be straighter perceptually. Conversely, unnatural videos that are straight in the intensity domain should be curved perceptually. Finally, natural videos that are straight in the intensity domain should remain straight perceptually. Our results are consistent with all three predictions, demonstrating that the visual system deploys nonlinear transformations targeted to straighten natural videos, in support of tasks that rely on prediction.