Published in Computational and Systems Neuroscience (CoSyNe), Feb 2017.
We compare several models of visual representation in terms of their ability to predict human judgements of visual distortion. Each model is defined by a differentiable mapping from image inputs to a response vector, which is then corrupted by additive Gaussian noise. We use the Fisher Information matrix to predict discrimination thresholds for the visibility of arbitrary distortions, up to an unknown scale factor. We test this by generating, for each model, a pair of specific distortions corresponding to the largest and smallest eigenvectors of the Fisher Information matrix, which represent the model-predicted most and least noticeable changes to the image, respectively. We distort the image by adding multiples of each vector, and measure detection thresholds for human subjects in a two-alternative forced-choice task. Results are quantified using the difference, D, of the log amplitude thresholds for detection of the two vectors. Two random perturbation vectors would yield of value of approximately D=0, and larger values of D indicate that the sensitivity of the model is better aligned with that of humans.
We used this methodology to test three models: a simple model of LGN neurons that includes local luminance and contrast normalization (Berardino et.al., cosyne2016; Mante et.al., 2008), and two different 4-stage convolutional neural networks. Each model was trained on a database of human perceptual judgments (Ponomarenko et.a., 2009). We found that, despite performing slightly better in terms of cross-validated correlation with the database, both artificial neural networks performed much worse than the LGN model on this test. We conclude that in this situation, where data are somewhat scarce, cross validation is not powerful enough to expose failures of a particular model class. Our method provides a more general form of cross-validation, based on the synthesis of model-optimized stimuli and comparison with human judgments on these targeted stimuli, and ensures that the model generalizes beyond curated data.