Denoising Diffusion Implicit Models

This is my personal notes following the DDIM paper and later on transcribed using chatGPT from screenshot of my note pages. There are a lot of mistakes but overall hopefully it still makes sense.

DDPM Revisit

Recall that DDPM defines a forward diffusion process (forward noising).

: original data

with , ,

or that

We also showed that we can sample directly from :

with

With , , so

What we actually want is the reverse process (denoising):

We show that this too is a Gaussian with the same variance and some mean. Our model learns to approximate this reverse process:

We learn by maximizing the likelihood:

or maximizing the log-likelihood

We show that it has a lowerbound

and we learn to maximize this lowerbound which in turn increases

The lower bound has a form:

where

Recall that we know is also a Gaussian with variance with some mean and hence we try to learn to match with:

But here we're dealing with which we can show to be:

with

(as , )

So is KL btw 2 Gaussians and:

So we figure out the left term of . For the right term, we have:

We can unify the 2 terms by predicting the noise instead.

We have:

Plug this into the equation of we have:

This means instead of predicting the mean of to match that of , we can predict the noise by , then:

Plug the new equations for and to , we have:

And since

and

So:

We can see that we can combine the left and right terms of into one:

with:

So training becomes:

1. Sample noise .

2. Sample

3. Predict noise .

4. Minimize

And for denoising, we go from . We start from

At any time step , we have and want to start from there to get . To do so we need , which we know to have variance and its mean is predicted by the network :

Then

Finally,

DDIM

DDIM paper wants to improve over DDPM, specifically to remove the need to denoise from .

The reason we need to go step by step from is because the forward process is Markovian. If we can find a different framework that avoids this characteristic, we might be able to denoise with less steps.

One key observation is the loss depends on the marginal only. They propose a new framework which eventually also depends only on . Interestingly, they propose to steer from the backward process instead. Starting with data , we sample the final :

(and of course, similar to DDPM, with , )

Now with fixed and , we "denoise":

Under this framework, we will have:

Indeed, we can prove this via induction.

Base case: is true by definition.

Assume it's true for , we need to prove it true for , that is:

We have:

From the definition of , we have:

The first term is fixed, and are Gaussian and independent, so easy to see is also Gaussian with

So DDIM has the same marginal as DDPM.

The forward process is then:

This forward process is no longer Markovian as it depends on both and . The paper also mentions this forward is also Gaussian but it's not important.

This is true as we shall see later that by defining the backward process already, we can already denoise (once trained), without any knowledge of the forward.

That is we start with and denoise using . We need to do so, we can predict it by using

As before we use a network to predict the noise:

So

The generative process is then:

We use Gaussian for so generative process is supported everywhere.

Using the same derivation as DDPM, we get the ELBO:

We have:

Since

Now for :

(KL of 2 Gaussians)

We have

So now putting this and the derivation for back to , we can see that maximizing equals to minimizing:

This is the same form as DDPM:

(Here's how I understand next.)

If parameters are not shared, the global optimum is independent of , meaning we might as well set , which is the case for DDPM training. And when , the training objectives for both DDPM and DDIM are the same.

In practice, if the model is large enough, it can learn different parameters for different time step. I guess that's why we can use the same network for all time step and achieve good performance.

Accelerated Sampling

Everything so far is only to show that using the same DDPM learning objective, we can train various models with different inference process as long as it leads to and depends on only the same marginals .

DDIM proposes another process that can generate using only a subset of steps.

Consider the subsequence of of length with , and let .

Define the backward:

The process for is the same as DDIM's we've discussed so far, while for the , they are sampled directly from using:

This way the inference process can be factorized as:

And the generative process is:

or equivalently,

By doing the same derivation for , we'll again obtain the same training objective function. The role of the "skipped" steps is there to make the loss match, but we don't use them for generation.

Let's look at the update sampling rules.

DDPM

DDIM

DDIM defines

When doing accelerated sampling, we replace with .

When doing the full steps and ,

When , DDIM is deterministic (no ).

When , the update rule looks more like DDPM.

Thoughts

The paper goes on to compare the performance of DDIM vs DDPM. A few key results:

• DDIM performs very well with much less number of sampling steps, like 10x less.
• DDPM with full sampling steps still performs the best, but DDIM is very close
• With , DDIM becomes deterministic as the only noise added is the predicted noise, no random noise.
• Deterministic DDIM is consistent. With the same , using different trajectories, i.e. different number of sampling steps, we get very similar images. This suggests contains information latent encoding of the image.
• Due to above reason, we can performs interpolation rather easily with DDIM.
• DDIM has low reconstruction noise.

I'm not going to discuss the results of the papers in detail, for I cannot do a better job than it does. Rather I'd like to play around with this new sampling strategy. What's really intriguing to me is that DDPM training is proposed for 1 strategy, but turn out under the hood it already supports various different family of generative processes. For example in accelerated sampling, just training DDPM already supports various sampling schedules (i.e. different ).

The use of predicted of DDIM is also interesting. When first learning DDPM, I wonder why we can't go directly to because after all we're predicting the noise added to to get . Well, of course we can't because it'll be very noisy. DDPM slowly denoises step by step. It also has to do so because that's how it is framed. For (deterministic) DDIM, the updating rule can be decomposed into 2 parts, one is "going towards the direction of (predicted) ", and the other is "still adding a bit of noise of ". The way I understand this is, we make a predict and commit to go there, but since we know it's very noisy so by adding back a bit of noise, it keeps around the area of . This way we move step by step to a mode of the less noisy data. Not sure if that makes sense...

Now I'm just going to play around with a trained DDPM but with DDIM sampling.

Code & Examples

I provide chatGPT with the updating rule for DDIM and ask it to add the implementation to an existing DDPM one. Using a pretrained DDPM checkpoint, I run a few experiments.

DDIM vs DDPM

Using a DDPM checkpoint trained with steps, I run denoising using DDPM/DDIM for 10 steps (beacause 50 is more than enough for MNIST, DDPM's output looks good in that case). As we can see, DDPM's output is still quite noisy yet DDIM's looks good.

I was curious so I trained another checkpoint but for only steps. Obviously it's a difficult task to denoise from pure gaussian to image in 50 steps so both outputs are terrible.

I was expecting to see worse results with high but overlook it looks good. This is mainly due to MNIST being a very easy task I think.

Number of steps

Here we experiment with using different number of denoising steps for DDIM. I also include results for DDPM. Obviously, more denoising steps works better.

Interpolation

My favorite part. Unfortunately I think it's hard to see the interpolation using MNIST. I'd like to think there's interpolation going on there. For example in the first column of DDIM, 2 and 0 interpolated results in something that looks like the original 2 but with a curve at the bottom that might come from 0? maybe, lol.