Langevin Sampling On Mixture of Disjoint Distributions

Note

After learning about Langevin Sampling, I continued to explore more about score matching. The book mentions this "...the Langevin procedure may not sample correctly if the data distribution comprises a mixture of disjoint distributions". This becomes the exercise 20.18 which we explore here. Assuming we have a distribution that:

And .

Langevin Sampling:

If ,

so

and therefore

Similarly for .

Meaning disappears from the score within each disconnected component. So even if, say, , Langevin sampling is not guaranteed to produce 90% of the samples from .

More generally, once and are disjoint, the score inside one component does not contain information about how much probability mass the other component should have. To solve this we bridge and so always appears. One way to do it is we use Parzen estimator to smear out the distribution. More specifically we'll use a Gaussian density kernel so that the probability will be > 0 everywhere and hence whatever comes from A will reach B and vice versa. We define:

with:

Therefore,

What this means is that for every original point , we create a noisy version , where . This replaces the original distribution with a smoothed distribution . The amount of smoothing depends on : larger spreads the probability mass farther away from each , while smaller keeps closer to . As , the Gaussian noise approaches a Dirac delta, so and the smoothed distribution approaches the original distribution .

For the mixture distribution,

so

and

Now we need to evaluate and so we can get the value of the score function.

For ,

Therefore,

For Gaussian noise,

Using

we have

or equivalently,

Therefore,

Similarly,

Now I have no idea how to proceed analytically from here. I think in general, the closed-form solution can be complicated or may not be convenient to derive.

In practice, if and are distributions that we can sample from, we can approximate these integrals using a large number of samples.

For example, draw

Then

can be approximated as

Since during Langevin sampling we are currently at , we simply evaluate

for every sampled , then average them to approximate .

We do the same thing for to approximate

We also approximate the smeared densities themselves:

and similarly for .

Then we plug everything back into

to get the score at the current point .

We can then use this score in the Langevin update to get .

Visualization

I asked chatGPT to write a script to visualize this smooth process and how it impacts sampling. The code can be found here. The video can be found: