Comparing image representations in terms of sensitivities to local distortions
D Lipshutz*, J Feather*, S Harvey, A Williams and E P Simoncelli
Published in Computational and Systems Neuroscience (CoSyNe), Mar 2025.
Similarity between neural representations is often quantified by measuring alignment of the representations over a set of natural stimuli that are relatively far apart in stimulus space (Kriegeskorte et. al. 2008). However, systems with similar global structure can have strikingly different sensitivities to local stimulus distortions (Szegedy et. al. 2013), suggesting a need for metrics that compare local sensitivities of representations. We propose a framework for comparing a set of image representations in terms of their sensitivities to local distortions. We quantify the local geometry of a representation using the Fisher information matrix (FIM), a standard statistical tool for characterizing the sensitivity to local stimulus perturbations, and use this to define a metric on the local geometry of representations near a base image. This metric may then be used to optimally discriminate a set of representations, by finding a pair of ``principal distortions'' (PDs) that maximize the variance of the representations under this metric. As an example, we apply our method to a set of deep neural network (DNN) models with training procedures designed to yield more human-aligned visual representations, and investigate whether the local geometries of the learned representations are more influenced by the DNN architecture or training procedure. More broadly, neuroscientists are often faced with the problem of selecting the most promising biological model from a long list of candidates. Experiments are expensive and time-limited, so it is critical to identify a minimal set of manipulations that distinguish the models. Our work provides a rigorous method for identifying such optimal stimulus perturbations, providing neuroscientists with actionable guidance for experiments that can probe the local geometry of neural representations.