Memories of the images that we have seen are thought to be reflected by in the reduction of neural responses in high-level visual brain areas such as inferotemporal (IT) cortex, a phenomenon known as repetition suppression (RS). We challenged this hypothesis with a task that required rhesus monkeys to report image familiarity while ignoring variations in contrast, a stimulus attribute that is also known to modulate the overall IT response. The monkeys' behavior was largely contrast-invariant, contrary to the predictions of the RS encoding scheme, which could not distinguish response familiarity from changes in contrast. However, the monkeys' behavioral patterns were well predicted by a linearly decodable variant in which the total spike count is corrected for contrast modulation. These results suggest that the IT neural activity pattern that best aligns with single-exposure visual familiarity behavior is not RS but rather "sensory referenced suppression" (SRS): reductions in IT population response magnitude, corrected for sensory modulation.