In this lecture, we will conclude our discussion of the standard multi-layer perceptron.
Please study the following material in preparation for the class:
- Hugo Larochelle’s video lectures 2.8 to 2.11.
- Chapters 6 of the Deep Learning textbook (end of Chapter 6: 6.4, 6.5, 6.6 – also review 6.2.3 and 6.2.4)
Other relevant material:
- Christopher Olah’s blog post on Neural Networks, Manifolds, and Topology (highly recommended!)
- Sigmoid versus Tanh and weight initialization: Understanding the difficulty of training deep feedforward neural networks. Xavier Glorot and Yoshua Bengio
- Rectified linear vs sigmoid, tanh: Deep Sparse Rectifier Neural Networks. Xavier Glorot, Antoine Bordes and Yoshua Bengio