NVIDIA‘s Revolutionary Noise2Noise AI Technique Removes Image Noise in Real-Time

Understanding Image Noise and Traditional Denoising Methods

Image noise is an unavoidable reality in digital photography. Whether it manifests as unsightly graininess, color speckling, or luminance distortions, noise can severely degrade image quality and obscure important details.

Noise arises from a variety of sources, both external and internal to the camera. External sources include low lighting conditions which require high ISO sensitivity or long exposures. Internally, noise results from imperfections and random fluctuations within the camera sensor itself as incoming light is converted to electrical signals.

Different manifestations of image noise. Source: Noise in Digital Image Processing

The most common types of noise encountered in real-world images are:

• Gaussian noise – Additive noise with a Gaussian distribution that is independent at each pixel. Gaussian noise is usually caused by high temperatures or ISO speeds.
• Shot noise – Noise due to the quantum nature of light and the uncertainty in photon arrival times. Shot noise has a Poisson distribution and increases with the square root of light intensity. It is the dominant noise in lighter image regions.
• Salt-and-pepper noise – Sparse light and dark distortions caused by analog-to-digital conversion errors or bit transmission errors. Dead or saturated sensor pixels can also appear as salt-and-pepper noise.

Traditional methods to combat image noise come with major drawbacks:

• Multi-exposure averaging – Taking multiple exposures of the same scene and averaging them reduces noise by a factor of sqrt(N) for N exposures. However, this requires a static scene, multiplies storage and bandwidth requirements, and can introduce ghosting artifacts from misalignment.
• Spatial filtering – Applying edge-preserving smoothing filters like bilateral filters or non-local means can reduce noise, but often at the cost of over-smoothing and loss of fine texture details. The filtering process is also computationally intensive.
• Frequency domain filtering – Transforming the image to the frequency domain with DCT or wavelets and suppressing high-frequency components can attenuate noise, but also blurs edges and loses high-frequency content in the underlying signal.

What‘s needed is a way to effectively remove image noise while preserving real edges and details, requiring only a single exposure, and running in real-time. NVIDIA‘s Noise2Noise technique delivers just that by leveraging the power of deep learning.

How Noise2Noise Conquers Image Noise with AI

NVIDIA researchers developed Noise2Noise as a revolutionary approach to image denoising powered by convolutional neural networks (CNNs). The key idea is to train a CNN to map from a noisy image to a less noisy version of the same image without ever using clean images.

This is a departure from traditional learning-based denoising methods which required supervised training on large datasets of noisy and corresponding clean images. Acquiring this paired data can be difficult and expensive, especially for applications like medical or scientific imaging where noiseless ground truth may be impossible to obtain.

Noise2Noise takes a self-supervised approach instead – the model is trained on pairs of independently corrupted versions of the same image. Remarkably, this is sufficient for the model to learn to remove the noise, thanks to a clever insight about the statistical properties of noise.

The self-supervised Noise2Noise training process. Source: Author.

Intuitively, the network learns to identify which parts of the image are consistent between the two noisy versions and therefore likely to be part of the underlying signal. The uncorrelated noise gets averaged out.

Mathematically, the researchers prove that training a CNN to predict one noisy version from another with L2 loss is equivalent to supervised training with clean targets, under the assumption of zero-mean noise. In expectation, the noise cancels out and the model converges to the same parameters as if it were trained with clean targets.

The Noise2Noise CNN architecture is based on the popular U-Net model. It features a contractive path of convolutional and pooling layers to extract image features at progressively coarser scales, and an expansive path of upsampling and convolutional layers to synthesize the denoised output. Skip connections propagate localized information from input to output.

The U-Net CNN architecture used in Noise2Noise. Source: U-Net: Convolutional Networks for Biomedical Image Segmentation

To handle diverse noise types and levels, the researchers trained Noise2Noise on the ImageNet dataset with a variety of synthetic noise models applied:

• Gaussian noise with standard deviations from 5 to 100
• Poisson noise from 5 to 50
• Bernoulli noise with drop probability 10% to 50%
• Salt-and-pepper and random-valued impulse noise with replacement probability 10% to 50%

A single Noise2Noise model was trained to remove all these noise types. Training took approximately 300 hours on NVIDIA Tesla P100 GPUs with the cuDNN-accelerated TensorFlow framework.

Evaluating Noise2Noise Performance and Robustness

Tested on held-out noisy images, the trained Noise2Noise model achieves remarkable denoising results. It effectively removes severe noise while preserving perceptually salient image structures. The mean squared error and peak signal-to-noise ratio (PSNR) metrics confirm significant quantitative improvements over the noisy input.

Noise2Noise achieves state-of-the-art denoising results on ImageNet. Source: Noise2Noise: Learning Image Restoration without Clean Data

What‘s more, Noise2Noise generalizes beyond the synthetic noise used in training. When applied to real-world noisy photographs from the RENOIR dataset, Noise2Noise improves the PSNR by over 6dB on average without any adaptation. This suggests the model has learned a robust prior over natural images that is effective at separating signal from a wide range of noise types.

Noise2Noise generalizes to unseen noise types and real noisy photographs. Source: Noise2Noise: Learning Image Restoration without Clean Data

Crucially, Noise2Noise runs in real-time. Denoising a 512×512 pixel image takes just 5 ms on an NVIDIA Titan X GPU, or around 40 ms on a mobile NVIDIA Jetson TX2 platform. This opens up the possibility of integrating Noise2Noise into camera viewfinder streams for live denoising preview.

Applications Across Photography, Science, and Medicine

The Noise2Noise denoising technique has extensive applications across fields where image noise is a limiting factor:

• Smartphone photography – Noise2Noise could be integrated directly into smartphone camera processing pipelines and photo editing apps. This would especially benefit low-light and night mode photography, enabling handheld exposures without flash.
• DSLR and mirrorless cameras – Noise2Noise could allow clean high ISO shooting to capture action in lower light without a tripod. Photographers could worry less about noise and focus more on composition and timing.
• Astrophotography – Noise is a major challenge for astrophotography, as subjects are inherently dim and faint. Noise2Noise could help reveal distant galaxies, nebulae, and other deep sky objects with unprecedented clarity.
• Medical and scientific imaging – Noise2Noise is especially promising for domains like microscopy, X-ray CT, PET, and MRI where dose fractionation and acquisition time are limited by radiation safety or patient comfort. Learned denoising could enable lower exposures without sacrificing diagnostic power.

Noise2Noise has wide-ranging applications across photography, astronomy, medical imaging, and more. Source: Author.

A New Paradigm of Self-Supervised Learning

Beyond the immediate application to denoising, Noise2Noise demonstrates the remarkable potential of self-supervised learning from corrupted data. This paradigm could be a game-changer for tasks where gathering clean ground truth is impractical, like image super-resolution, motion deblurring, compression artifact removal, and more.

The key is to leverage the statistical properties of the corruption model, as Noise2Noise does by exploiting the zero-mean nature of noise. Extensions to Noise2Noise have shown that this approach can be generalized to handle spatially correlated noise and other structured image corruptions.

Self-supervised models also have the advantage of learning priors directly from real-world data, potentially outperforming supervised models trained on limited synthetic data. This opens up new avenues for using large unlabeled datasets to train models that are robust to the diverse artifacts encountered in the wild.

Future Research Directions and Integration into Photography Workflows

Noise2Noise is just the beginning of learned denoising. Future research may yield further improvements in quality, generalization, and efficiency:

• Training on an even wider range of real noise types and levels, including noise from different camera sensors
• Optimizing CNN architectures and loss functions for denoising, e.g. perceptual, adversarial, or texture losses
• Combining self-supervised and semi-supervised methods when some clean targets are available
• Exploring Bayesian approaches to estimate uncertainty and adapt to noise at test time
• Accelerating inference with network pruning, low-bit quantization, or custom hardware

As the Noise2Noise technique matures, we can expect to see it deployed at scale to improve the photos and videos we capture every day. Smartphone makers may license Noise2Noise to provide built-in denoising for all camera apps. Social media platforms may use Noise2Noise to enhance noisy low-light images uploaded by users. Pro-grade cameras may optimize Noise2Noise for larger image sensors and high-res video. Photo editing software may offer one-click Noise2Noise to salvage underexposed shots.

One day, we may look back in disbelief at the noisy images we once simply accepted as inevitable. Advances like NVIDIA‘s Noise2Noise are ushering in a future where stunning image clarity is the norm. By harnessing the power of AI, we are overcoming the physical limitations of camera sensors and achieving superhuman levels of perception. The age of noise-free photography is dawning.