Generalization in diffusion models arises from geometry-adaptive harmonic representation

Z Kadkhodaie, F Guth, E P Simoncelli and S Mallat

arXiv.org e-prints, Technical Report 2310.02557, Oct 2023.

DOI: 10.48550/arXiv.2310.02557

This paper has been superseded by:
Generalization in diffusion models arises from geometry-adaptive harmonic representation
Z Kadkhodaie, F Guth, E P Simoncelli and S Mallat.
Int'l Conf on Learning Representations (ICLR), vol.12 May 2024.
Recipient, Outstanding Paper award.


Download:

  • Reprint (pdf)
    • High-quality samples generated with score-based reverse diffusion algorithms provide evidence that deep neural networks (DNN) trained for denoising can learn high-dimensional densities, despite the curse of dimensionality. However, recent reports of memorization of the training set raise the question of whether these networks are learning the "true" continuous density of the data. Here, we show that two denoising DNNs trained on non-overlapping subsets of a dataset learn nearly the same score function, and thus the same density, with a surprisingly small number of training images. This strong generalization demonstrates an alignment of powerful inductive biases in the DNN architecture and/or training algorithm with properties of the data distribution. We analyze these, demonstrating that the denoiser performs a shrinkage operation in a basis adapted to the underlying image. Examination of these bases reveals oscillating harmonic structures along contours and in homogeneous image regions. We show that trained denoisers are inductively biased towards these geometry-adaptive harmonic representations by demonstrating that they arise even when the network is trained on image classes such as low-dimensional manifolds, for which the harmonic basis is suboptimal. Additionally, we show that the denoising performance of the networks is near-optimal when trained on regular image classes for which the optimal basis is known to be geometry-adaptive and harmonic.
  • Related Publications: Kadkhodaie22a, Kadkhodaie21a, Kadkhodaie20a, MohanKadkhodaie19b, Raphan10
  • Listing of all publications