Generative Models

Overview

Generative Neural Networks are architectures designed to learn and generate new data samples that resemble a given training distribution. These models can create synthetic data, ranging from images and text to music and more.

structure of GANs

Key aspects:

Distribution learning
Sample generation
Latent space modeling
Conditional generation

Core Concepts

Generative Adversarial Networks (GANs)
▶
Two-network adversarial training:
- Generator network
- Discriminator network
- Adversarial loss
- Mode collapse
$$ \min_G \max_D V(D,G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))] $$ $$ L_G = -\mathbb{E}_{z \sim p_z}[\log D(G(z))] $$ $$ L_D = -\mathbb{E}_{x \sim p_{data}}[\log D(x)] - \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))] $$
Flow-based Models
▶
Invertible transformations for exact likelihood:
- Bijective mappings
- Change of variables
- Normalizing flows
- Exact likelihood
$$ \log p_X(x) = \log p_Z(f(x)) + \log |\det J_f(x)| $$ $$ z = f(x) $$ $$ x = f^{-1}(z) $$ $$ J_f = df/dx $$
Diffusion Models
▶
Gradual noise addition and removal:
- Forward diffusion
- Reverse diffusion
- Score matching
- Denoising
For the full mathematical formulation of the diffusion model, see the original DDPM paper: Denoising Diffusion Probabilistic Models (Ho et al., 2020).

Implementation

PyTorch GAN and WGAN

▶

Example implementations of a basic GAN and Wasserstein GAN (WGAN) using PyTorch:


import torch
import torch.nn as nn

class Generator(nn.Module):
    def __init__(self, latent_dim, hidden_dims, output_dim):
        super(Generator, self).__init__()
        layers = []
        prev_dim = latent_dim
        for hidden_dim in hidden_dims:
            layers.extend([
                nn.Linear(prev_dim, hidden_dim),
                nn.BatchNorm1d(hidden_dim),
                nn.ReLU(inplace=True)
            ])
            prev_dim = hidden_dim
        layers.append(nn.Linear(prev_dim, output_dim))
        layers.append(nn.Tanh())
        self.model = nn.Sequential(*layers)
    def forward(self, z):
        return self.model(z)

class Discriminator(nn.Module):
    def __init__(self, input_dim, hidden_dims):
        super(Discriminator, self).__init__()
        layers = []
        prev_dim = input_dim
        for hidden_dim in hidden_dims:
            layers.extend([
                nn.Linear(prev_dim, hidden_dim),
                nn.LeakyReLU(0.2, inplace=True),
                nn.Dropout(0.3)
            ])
            prev_dim = hidden_dim
        layers.append(nn.Linear(prev_dim, 1))
        layers.append(nn.Sigmoid())
        self.model = nn.Sequential(*layers)
    def forward(self, x):
        return self.model(x)

class WGAN(nn.Module):
    def __init__(self, latent_dim, hidden_dims, output_dim, critic_iters=5, clip_value=0.01):
        super(WGAN, self).__init__()
        self.generator = Generator(latent_dim, hidden_dims, output_dim)
        self.critic = Discriminator(output_dim, hidden_dims)
        self.latent_dim = latent_dim
        self.critic_iters = critic_iters
        self.clip_value = clip_value
    def clip_critic_weights(self):
        for p in self.critic.parameters():
            p.data.clamp_(-self.clip_value, self.clip_value)
    def train_step(self, real_data, optimizer_g, optimizer_c):
        batch_size = real_data.size(0)
        critic_loss = 0
        for _ in range(self.critic_iters):
            z = torch.randn(batch_size, self.latent_dim)
            fake_data = self.generator(z)
            critic_real = self.critic(real_data)
            critic_fake = self.critic(fake_data.detach())
            critic_loss = -(torch.mean(critic_real) - torch.mean(critic_fake))
            optimizer_c.zero_grad()
            critic_loss.backward()
            optimizer_c.step()
            self.clip_critic_weights()
        z = torch.randn(batch_size, self.latent_dim)
        fake_data = self.generator(z)
        critic_fake = self.critic(fake_data)
        generator_loss = -torch.mean(critic_fake)
        optimizer_g.zero_grad()
        generator_loss.backward()
        optimizer_g.step()
        return {
            'critic_loss': critic_loss.item(),
            'generator_loss': generator_loss.item()
        }

NumPy Normalizing Flow (Sketch)

▶

Manual sketch of a normalizing flow step in NumPy:


import numpy as np

def affine_flow_forward(x, scale, shift):
    z = scale * x + shift
    log_det_jacobian = np.log(np.abs(scale))
    return z, log_det_jacobian

def affine_flow_inverse(z, scale, shift):
    x = (z - shift) / scale
    return x
# Example usage:
x = np.random.randn(10)
scale = 2.0
shift = 1.0
z, log_det = affine_flow_forward(x, scale, shift)
x_recon = affine_flow_inverse(z, scale, shift)

Interview Examples

Mode Collapse in GANs

Explain mode collapse in GANs. What are some techniques to mitigate it?

# Key points for interview:
# - Mode collapse: Generator produces limited variety of outputs (few modes of data distribution).
# - Mitigation: Use minibatch discrimination, unrolled GANs, feature matching, Wasserstein loss, or improved training techniques.

                    

Implement a Simple GAN Training Loop (PyTorch)

Write a basic GAN training loop in PyTorch, including generator and discriminator updates.

import torch
import torch.nn as nn
import torch.optim as optim

def train_gan(generator, discriminator, data_loader, latent_dim, num_epochs=1):
    criterion = nn.BCELoss()
    optimizer_g = optim.Adam(generator.parameters(), lr=0.0002)
    optimizer_d = optim.Adam(discriminator.parameters(), lr=0.0002)
    for epoch in range(num_epochs):
        for real_data, _ in data_loader:
            batch_size = real_data.size(0)
            real_labels = torch.ones(batch_size, 1)
            fake_labels = torch.zeros(batch_size, 1)
            # Train discriminator
            z = torch.randn(batch_size, latent_dim)
            fake_data = generator(z)
            outputs_real = discriminator(real_data)
            outputs_fake = discriminator(fake_data.detach())
            d_loss = criterion(outputs_real, real_labels) + criterion(outputs_fake, fake_labels)
            optimizer_d.zero_grad()
            d_loss.backward()
            optimizer_d.step()
            # Train generator
            z = torch.randn(batch_size, latent_dim)
            fake_data = generator(z)
            outputs = discriminator(fake_data)
            g_loss = criterion(outputs, real_labels)
            optimizer_g.zero_grad()
            g_loss.backward()
            optimizer_g.step()

                    

Practice Questions

1. Explain the core concepts of Generative Models Easy

Hint: Think about the fundamental principles

2. How would you implement this in a production environment? Hard

Hint: Consider scalability and efficiency

3. What are the practical applications of Generative Models? Medium

Hint: Consider both academic and industry use cases

Generative Models

Overview

Core Concepts

Generative Adversarial Networks (GANs)

Flow-based Models

Diffusion Models

Implementation

PyTorch GAN and WGAN

NumPy Normalizing Flow (Sketch)

Interview Examples

Mode Collapse in GANs

Implement a Simple GAN Training Loop (PyTorch)

Practice Questions

1. Explain the core concepts of Generative Models Easy

2. How would you implement this in a production environment? Hard

3. What are the practical applications of Generative Models? Medium

Related Resources

Related Topics