In this lecture we will continue our discussion of probabilistic undirected graphical models with the** Deep Belief Network**. The material is that listed from the last lecture plus the material below.

**Please study the following material in preparation for the class:**

- Chapter 15 (sec. 15.1) of the Deep Learning Textbook (estimating the partition function.
- Texture modeling slides

**Other relevant material:**

- Texture Modeling with Convolutional Spike-and-Slab RBMs and Deep Extensions Heng Luo, Pierre Luc Carrier, Aaron Courville, Yoshua Bengio. This is an application of the deep belief network as a generative model.

Advertisements

The link for Chapter 15 actually points to Chapter 21, and I can’t find the list of hyperlinked chapters 😦

LikeLike

http://www.iro.umontreal.ca/~bengioy/dlbook/partition.html

LikeLike

Sorry about that. The link should now be fixed.

LikeLike

Both AIS and bridge sampling provide a means to estimate the ratio between Z1 and Z0, but the whole point of this is to estimate Z1, which is intractable. Am I right in assuming that p0 must be designed to make the computation of Z0 tractable, in order to estimate Z1?

Also, in section 15.1.3, there’s a mention of an RBM’s “temperature”. What’s an RBM’s temperature?

LikeLike

Regarding your second question, basically, when we have multimodal distribution, Gibbs Sampling fails to explore modes far away from initial sample. This is a more severe problem in CD which the sampling chain is short.

Several methods have been proposed to improve “mixing property” of Gibbs Sampling. There are some “Temperature” based methods by Toronto people such as Parallel Tempering.

In a nutshell, temperature denotes the overall energy of the system. For example in RBM, they easily multiply energy by inverse of temperature (B):

P(v, h) = exp{-B E(v, h)}/Z

Basically, the higher the temperature, the more likely the samples to go far away.

LikeLiked by 1 person

Are there any standard ways of choosing the starting point of the Gibbs sampler?

LikeLike