Representations of image memory and image memorability are largely non-overlapping in inferotemporal cortex

V Mehrpour, E P Simoncelli and NC Rust.

Published in Annual Meeting, Neuroscience, Oct 2019.

This paper has been superseded by:
Pinpointing the neural signatures of single-exposure visual recognition memory
V Mehrpour, T Meyer, E P Simoncelli and N C Rust.
Proc. Nat'l Academy of Sciences, vol.118(18), May 2021.


Humans are generally very good at remembering the images they have seen, although some images are easier to remember than others. Neural correlates of both 'image memory' and 'image memorability' have been linked to inferotemporal cortex (IT), but current accounts of their representations are in conflict. Considerable evidence suggests that image memory is signaled by a reduction in response for familiar as compared to novel images ('repetition suppression'), and lower firing rates correspond to higher behavioral memory performance. in contrast, recent findings demsontrate that more memorable images evoke stronger responses in IT, both when images are novel and when they are repeated as familiar, thereby viiolating the tenet that lower firing rates map to more robust memory behavior. Is there any way to resolve these conflicting accounts of image memory and image memorability representations in IT? To address this question, we analyzed neural data collected from IT as two rhesus monkeys performed a single-exposure visual memory task in which they viewed images and indicated whether they were novel (never seen before) or familiar (seen exactly once). We hypothesized that both image memory and image memorability representations occupied multidimensional subspaces in IT that are partially overlapping, and that the region of overlap includes that occupied by a total spike count coding scheme. To test this hypothesis, we compared cross-validated linear decoding performance for memory and/or memorability when applied to the intact data and after removing different types of information from the population. Iteratively projecting out mulitple orthogonal dimensions of each type (memory or memorability) was required to obliterate decoding performance for the same type, consistent with the hypothesis that both memory and memorability representations are multi-dimensional in IT. Projecting out information that fell along the total spike count vector (i.e., 1,1,1,...) had a measurable albeit small impact on both memory and memorability performance, consistent with the hypothesis that both memory and memorability subspaces overlap in the region occupied by the total spike count decoder. Finally, projecting the linear subspace for one type of information (memory or memorability) had minimal impact on the performance of the other type of decoder beyond that attributed to the total spike count. Together, these results suggest that iamge memory and image memorability occupy partially but largely non-overlapping linear subspaces in IT.
  • Listing of all publications