A Detailed Guide to Generative Adversarial Networks (GANs) with TensorFlow Implementation

Generative adversarial networks (GANs) are one of the most exciting recent innovations in deep learning. First introduced by Ian Goodfellow et al. in 2014, GANs are a class of generative models that can create new data (such as images) that resemble your training data. They have achieved impressive results in tasks such as generating photorealistic images, converting images from one domain to another (e.g. day to night, sketches to photos), and much more.

In this guide, we‘ll provide a detailed explanation of how GANs work, walk through implementing a basic GAN in TensorFlow and Keras, and discuss some amazing real-world applications of this powerful technique. By the end, you‘ll understand the core concepts behind GANs and be equipped to start applying them to your own projects. Let‘s dive in!

What are Generative Adversarial Networks?

At their core, GANs are a type of generative model—a model that can generate new data similar to what it was trained on. Specifically, GANs are based on a novel adversarial training framework in which two neural networks—the generator and discriminator—compete against each other.

The generator network takes random noise as input and learns to generate synthetic samples that approximate the distribution of real data. The discriminator network learns to distinguish between the real samples from the training set and the synthetic samples created by the generator. During training, the generator aims to fool the discriminator into misclassifying its outputs as real, while the discriminator aims to correctly identify the fakes. This adversarial dynamic forces the networks to continuously improve and results in the generator producing increasingly realistic data.

The GAN training process involves the generator and discriminator networks competing against each other. (Source: Skymind)

Mathematically, the GAN framework corresponds to a minimax two-player game in which we aim to find the Nash equilibrium point—a point at which neither player can improve by changing strategy. Using the notation from Goodfellow‘s original GAN paper:

• Let pdata(x) be the distribution of real data
• Let pz(z) be the input noise distribution for the generator
• Let G(z; θg) be the generator network with parameters θg
• Let D(x; θd) be the discriminator network with parameters θd

Then the minimax objective function for GAN training is:

The generator aims to minimize this loss function, while the discriminator tries to maximize it. In the inner term, the discriminator wants to correctly classify real samples as real (by outputting D(x) close to 1) and generated samples as fake (by outputting D(G(z)) close to 0). The outer term shows that the generator wants to generate samples that fool the discriminator into outputting 1.

Through many rounds of this adversarial game, with the generator and discriminator networks iteratively updating their weights, the system reaches an equilibrium. At this point, the generator produces samples that closely match the real data distribution, and the discriminator can no longer distinguish the real and fake samples.

Implementing a Basic GAN in TensorFlow and Keras

Now that we understand the high-level GAN framework, let‘s walk through implementing a simple GAN to generate handwritten digits using the MNIST dataset and TensorFlow/Keras. We‘ll use a deep convolutional GAN (DCGAN) architecture.

First, let‘s define the generator network, which will take 100-dimensional noise vectors as input and output 28×28 pixel images:

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((7, 7, 256)))
    assert model.output_shape == (None, 7, 7, 256) 

    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding=‘same‘, use_bias=False))
    assert model.output_shape == (None, 7, 7, 128)  
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding=‘same‘, use_bias=False))
    assert model.output_shape == (None, 14, 14, 64)    
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding=‘same‘, use_bias=False, activation=‘tanh‘))
    assert model.output_shape == (None, 28, 28, 1)

    return model

This generator uses transpose convolutions to progressively upsample the feature maps into a final 28×28 image. Batch normalization and leaky ReLU activations are used to stabilize training. The final layer uses a tanh activation to keep the output in the [-1, 1] range, matching the normalized MNIST pixels.

Next, we define the discriminator network, which is a standard binary classifier convolutional network:

def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding=‘same‘, 
                                     input_shape=[28, 28, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding=‘same‘))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model  

The discriminator downsamples the input image through strided convolutions, flattens the feature maps, and outputs a single probability value indicating whether the input is real or fake.

With the generator (G) and discriminator (D) networks defined, we can specify the training procedure, iterating between training D on real and fake samples and then training G to fool D:

@tf.function
def train_step(images):
    noise = tf.random.normal([BATCH_SIZE, noise_dim])

    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
      generated_images = generator(noise, training=True)

      real_output = discriminator(images, training=True)
      fake_output = discriminator(generated_images, training=True)

      gen_loss = generator_loss(fake_output)
      disc_loss = discriminator_loss(real_output, fake_output)

    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)
    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)

    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))
    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))

The generator loss is defined as:

def generator_loss(fake_output):
    return cross_entropy(tf.ones_like(fake_output), fake_output)

This captures the generator‘s objective of having D classify its samples as real (represented by the target label of 1).

The discriminator loss is:

def discriminator_loss(real_output, fake_output):
    real_loss = cross_entropy(tf.ones_like(real_output), real_output)
    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)
    total_loss = real_loss + fake_loss
    return total_loss

Reflecting the discriminator‘s goal of correctly classifying real samples as real and fake samples as fake.

We use the Adam optimizer and binary cross-entropy loss for both networks. Here is the full training loop, which loads the MNIST data and trains the GAN for a specified number of epochs, logging the losses and saving sample outputs periodically:

def train(dataset, epochs):
  for epoch in range(epochs):
    start = time.time()

    for image_batch in dataset:
      train_step(image_batch)

    display.clear_output(wait=True)
    generate_and_save_images(generator,
                             epoch + 1,
                             seed)

    if (epoch + 1) % 15 == 0:
      checkpoint.save(file_prefix = checkpoint_prefix)

    print (‘Time for epoch {} is {} sec‘.format(epoch + 1, time.time()-start))

  display.clear_output(wait=True)
  generate_and_save_images(generator,
                           epochs,
                           seed)

After training this simple DCGAN on MNIST for 50 epochs, we can generate convincing fake handwritten digits:

The generated samples capture the style and diversity of the real MNIST data. This demonstrates the power of the GAN framework—with a relatively straightforward architecture and training process, we can synthesize highly realistic data!

Applications of GANs

Since their introduction in 2014, GANs have been successfully applied to a wide range of tasks in computer vision, natural language processing, and beyond. Some exciting applications include:

• Generating photorealistic images of faces, objects, and scenes
• Converting sketches or semantic labels to realistic images
• Transferring images between domains, such as day to night or summer to winter
• Synthesizing 3D object models
• Enhancing low-resolution images with realistic high-resolution details
• Generating realistic samples of speech and music
• Producing synthetic tabular data that preserves privacy

And many more! Active research continues to push the boundaries of what GANs can achieve.

To highlight a couple powerful examples, check out NVlabs‘ StyleGAN architecture that can generate shockingly photorealistic portraits of people who don‘t exist:

StyleGAN can generate eerily realistic photos of humans who don‘t exist (Source: NVlabs)

Or GANs being used to transform satellite photos to maps and vice versa!

The Pix2Pix GAN architecture enables cross-domain image translation, like converting between satellite images and maps (Source: Pix2Pix)

These amazing results only scratch the surface of the huge range of exciting applications of generative adversarial networks. With their power to synthesize highly realistic data, GANs open up transformative possibilities across domains—from generating synthetic training data to creative media manipulation and far beyond.

Further GAN Architectures and Techniques

The basic DCGAN implemented above is a great starting point for understanding GANs, but researchers have developed many more sophisticated architectures and techniques to improve training stability and generated output quality. A few key examples:

• Wasserstein GANs (WGANs) – Use the Wasserstein distance metric to measure how different the real and generated distributions are, enabling more stable and robust convergence compared to regular GANs
• Conditional GANs – Incorporate additional label information in both the generator and discriminator to produce outputs conditioned on the label (for example, generating a specified digit class for MNIST)
• CycleGAN – Trains two GANs to translate images from domain X to Y and Y to X, with additional "cycle consistency" losses to ensure translating from X -> Y -> X reproduces the original image
• Progressively-Growing GANs – Trains generator and discriminator by gradually increasing the image resolution, allowing learning to first focus on large-scale structure before refining fine details
• Self-Attention GANs (SAGANs) – Incorporates self-attention layers to enable the networks to model long-range dependencies across image regions, improving quality
• BigBiGAN – Introduces an additional encoder network trained to learn an embedding space for generated images that can also be used for downstream supervised tasks

This is just a sampling of the many GAN variants that have been developed. The GAN framework is highly flexible and continues to be a very active area of research.

Conclusion

Generative adversarial networks are an exciting and rapidly developing class of generative models. By pitting a generator and discriminator network against each other in an adversarial game, GANs can produce highly realistic synthetic data and unlock powerful applications across a huge range of domains.

In this guide, we:

• Explained the core GAN framework and adversarial training process
• Discussed Nash equilibrium and the mathematical minimax GAN objective
• Walked through implementing a DCGAN to generate MNIST digits in TensorFlow and Keras
• Highlighted amazing applications like generating photorealistic images and converting between image domains
• Overviewed more advanced GAN architectures and techniques

I hope this has given you a solid understanding of how GANs work and equipped you to start applying this incredible technology yourself. The field of generative modeling with GANs is progressing rapidly, and we can‘t wait to see what you‘ll build!

Thanks for reading! Let me know if you have any other questions.