Lecture 18, March 19th, 2015: Deep Belief Networks

In this lecture we will continue our discussion of probabilistic undirected graphical models with the Deep Belief Network. The material is that listed from the last lecture plus the material below.

Please study the following material in preparation for the class:

Other relevant material:


6 Replies to “Lecture 18, March 19th, 2015: Deep Belief Networks”

  1. Both AIS and bridge sampling provide a means to estimate the ratio between Z1 and Z0, but the whole point of this is to estimate Z1, which is intractable. Am I right in assuming that p0 must be designed to make the computation of Z0 tractable, in order to estimate Z1?

    Also, in section 15.1.3, there’s a mention of an RBM’s “temperature”. What’s an RBM’s temperature?


    1. Regarding your second question, basically, when we have multimodal distribution, Gibbs Sampling fails to explore modes far away from initial sample. This is a more severe problem in CD which the sampling chain is short.
      Several methods have been proposed to improve “mixing property” of Gibbs Sampling. There are some “Temperature” based methods by Toronto people such as Parallel Tempering.
      In a nutshell, temperature denotes the overall energy of the system. For example in RBM, they easily multiply energy by inverse of temperature (B):

      P(v, h) = exp{-B E(v, h)}/Z

      Basically, the higher the temperature, the more likely the samples to go far away.

      Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s