Comparing the local information geometry of image representations

D Lipshutz*, J Feather*, S E Harvey, A H Williams and E P Simoncelli.

Published in Proc. UniReps: Unifying Representations in Neural Models, A workshop at Neural Information Processing Systems (NeurIPS), Dec 2024.

This paper has been superseded by:
Discriminating image representations with principal distortions
J Feather*, D Lipshutz*, S E Harvey, A H Williams and E P Simoncelli.
Int'l Conf on Learning Representations (ICLR), vol.13 Apr 2025.

We propose a framework for comparing a set of image representations (artificial or biological) in terms of their sensitivities to local distortions. We quantify the local geometry of a representation using the Fisher information matrix (FIM), a standard statistical tool for characterizing the sensitivity to local distortions of a stimulus, and use this as a substrate for a metric on the local geometry of representations in the vicinity of a base image. This metric may then be used to optimally differentiate a set of models, by optimizing for a pair of distortions that maximize the variance of the models under this metric. We use the framework to compare a set of simple models of the early visual system, identifying a novel set of image distortions that allow immediate comparison of the models by visual inspection. In a second example, we show that the method can reveal distinctions between standard and adversarially trained object recognition networks.
  • Listing of all publications