My note on the last part of chapter 20 of "Deep Learning" book by Bishop & Bishop.

Guided Diffusion

It's easier to talk about class-conditional diffusion through the lens of score-matching. Instead of using the score function , we can use . Using Bayes theorem:

The second term guides the denoising process towards the direction that maximizes given a class .

We can further control this with a hyperparameter :

The issue with this approach is:

• we gotta train a classifier
• we gotta train it with various amount of noise levels applied on

Classifier-free approach fixes this by substituting with .

Based on the similarity between score matching and DDPM, we can see that we can do the same thing with DDPM. That is, the network becomes . During training, we can learn both the unconditional and conditional predictions using the same model: use a null label for and label for . With the null label, the model learns the score/noise prediction for the overall data distribution, while with , it learns the prediction conditioned on samples belonging to class . In practice, we use a dropout probability to randomly choose whether to use the null class or during training.

Based on the similarity between score matching and DDPM, we can see that we can do the same thing with DDPM. That is, the network becomes . During inference, we run the network twice: once with the null class and once with , then combine their outputs based on the equation above.

We can think of the unconditional prediction as pointing along the denoising trajectory toward samples from the overall data distribution, without caring about which class they belong to. On the other hand, the conditional prediction points toward samples belonging to class . When , since , we are no longer simply interpolating between these two predictions. Instead, we extrapolate beyond the conditional prediction by subtracting some of the unconditional direction. Intuitively, this emphasizes the part of the conditional prediction that specifically pushes the sample toward class , giving us stronger class guidance.

Using the same codebase for ddpm, we add support for class conditioning, see here. See samples here. Overall it works very well for MNIST.