Unsupervised deep video denoising

D Y Sheth*, S Mohan*, J L Vincent, R Manzorro, P A Crozier, M M Khapra, E P Simoncelli and C Fernandez-Granda

arXiv.org e-prints, Technical Report 2011.15045, Dec 2020.

DOI: 10.48550/arXiv.2011.15045

This paper has been superseded by:
Unsupervised deep video denoising
D Y Sheth*, S Mohan*, J L Vincent, R Manzorro, P A Crozier, M M Khapra, E P Simoncelli and C Fernandez-Granda.
Int'l Conf. Computer Vision (ICCV), Oct 2021.


Download:

  • Reprint (pdf)
  • Examples, Software {PyTorch}
    • Deep convolutional neural networks (CNNs) currently achieve state-of-the-art performance in denoising videos. They are typically trained with supervision, minimizing the error between the network output and ground-truth clean videos. However, in many applications, such as microscopy, noiseless videos are not available. To address these cases, we build on recent advances in unsupervised still image denoising to develop an Unsupervised Deep Video Denoiser (UDVD). UDVD is shown to perform competitively with current state-of-the-art supervised methods on benchmark datasets, even when trained only on a single short noisy video sequence. Experiments on fluorescence-microscopy and electron-microscopy data illustrate the promise of our approach for imaging modalities where ground-truth clean data is generally not available. In addition, we study the mechanisms used by trained CNNs to perform video denoising. An analysis of the gradient of the network output with respect to its input reveals that these networks perform spatio-temporal filtering that is adapted to the particular spatial structures and motion of the underlying content. We interpret this as an implicit and highly effective form of motion compensation, a widely used paradigm in traditional video denoising, compression, and analysis. Code and iPython notebooks for our analysis are available in this URL.
  • Listing of all publications