Published in Annual Meeting, Neuroscience, Oct 2019.
Sensory systems make predictions about future observations given the recent past, and we have recently hypothesized that this is achieved by transforming them to an internal representation that follows straighter temporal trajectories (the "straightening hypothesis", Hénaff et al., 2018; 2019). This hypothesis is supported by both perceptual (Hénaff et al., 2018; 2019) and neural (Bai et al., 2018) evidence, which reveal straightening of natural image sequences, but increased curvature ("entangling") of artificial sequences that fade between an initial and final frame. Straightening of natural video trajectories, as well as entangling of artificial sequences, can also be observed in the responses of a two-stage model mimicking the nonlinear functional properties of the early visual system. Artificial neural networks trained for object recognition, on the other hand, exhibit an increase in curvature for all sequences. Thus, optimizing for object recognition does not provide these networks with the straightening capabilities found in the human visual system (Hénaff et al., 2019). To further investigate the straightening performance of artificial neural networks, we examined the responses of a biologically-inspired network optimized for image compression (Ballé, et al., 2017). The network consists of three stages, each computing responses of a set of (learned) linear filters, and normalizing their responses by a weighted combination (also learned) of other rectified filter responses. We find that this image compression network can achieve temporal straightening behaviors not seen in object recognition-trained networks, and more consistent with those of human observers. Specifically, we find that the image compression network exhibits straightening of natural sequences, as well as tangling of unnatural sequences. We conclude that the temporal straightening capabilities of primate visual systems are consistent with, and may depend on, nonlinear response properties found in biological neurons.