Office hour

Sorry for the late notice, as I mentioned yesterday, I’m making myself available between 12-1pm for general questions in reparation of the exam on Monday.

Presentations Thursday, April 9th, 2015

As previously announced, Thursday will be our presentation day.

Please make sure to send me your presentations (pdf format, 3-4 slides) by Wednesday 8pm at the latest.

Presentations should be about 4 minutes + time for one question (30 secs or so).

See you all Thursday!

 

Important upcoming dates

As we discussed in class yesterday, here are the important dates for the remainder of our course.

  • April 9th, 2015: Spotlight project presentations. The presentations should each be a max of 4 minutes (3 slides max.) and should highlight your most significant contributions.
  • April 13th, 2015: Final exam. 14h30-17h30 (3hours). Same room as our regular Monday class (Z210). Closed book but you are allowed 1 8.5×10 “cheat-sheet” (both sides)  for notes. The notes must be hand-written.
  • April 20th, 2015: Course Project due date. All project blog should be upto date and finalized by this date.

Quiz: March 9th, 2015

As announced in class yesterday, we will have a quiz on March 9th. You will have the full 2 hours for the quiz and it will be “open book” meaning that all material is available to you.

It will be worth 3% of your final grade (taken out of the 30% participation component).

Good luck.

 

Leaderboard for the class project

You are encouraged to publicize your results on the class project through the newly-created leaderboard wiki, which you can find here.

The wiki is editable by anybody with a Github account, so feel free to add your results by yourself. Here are the instructions, taken from the wiki:

Use this wiki to publicize your results on the Dogs vs. Cats class project.

Every time you get a better result on the challenge, you can insert an entry to the list below (at the appropriate place please, so that the list stays sorted by test error rate). Make sure to include a link to a blog post detailing how you achieved that result. The format for entries is as follows:

<test error rate> (<train error rate>, <valid error rate>): <short description> (<link to blog post>)

Class project starts now!

The Pylearn2 implementation of the Dogs vs. Cats dataset is complete, which means that you now have everything you need to start training models for the class project.

This post details where to find and how to use the dataset implementation along with Pylearn2.

Requirements

  • Pylearn2 and its dependencies
  • PyTables

Getting the code

You can find the code for the dataset in Vincent Dumoulin’s repository.

Clone the repo in a place that’s listed in your PYTHONPATH, and you’re ready to go.

Getting the data

N.B.: This part is only required if you’re working on your own machine. The data is already available on the LISA filesystem.

Downloading the images

First, you’ll need to make sure your PYLEARN2_DATA_PATH environment variable is set (e.g. through an export PYLEARN2_DATA_PATH=<path_of_your_choice> call in your .bashrc if you’re on Linux). This is where Pylearn2 expects your data to be found.

Create a dogs_vs_cats directory in ${PYLEARN2_DATA_PATH}.

Finally, download and unzip the train.zip file under ${PYLEARN2_DATA_PATH}/dogs_vs_cats directory (many thanks to Kyle Kastner for making it available without having to go through the whole Kaggle signup process!)

Generating the HDF5 dataset file

Once the images have been downloaded, unzipped and placed into the ${PYLEARN2_DATA_PATH}/dogs_vs_cats directory, run

python ift6266h15/code/pylearn2/datasets/generate_dogs_vs_cats_dataset.py

This will create an HDF5 file under ${PYLEARN2_DATA_PATH}/dogs_vs_cats/train.h5 which contains the whole training set. This may take some time. The file should weight around 11 gigabytes.

Instantiating and iterating over the dataset

We’re going to use the ift6266h15.code.pylearn2.datasets.variable_image_dataset.DogsVsCats subclass of Dataset.

The dataset constructor expects three arguments: an instance of a ift6266h15.code.pylearn2.dataset.variable_image_dataset.BaseImageTransformer subclass, and optionally a starting and stopping index specifying what slice of the whole dataset to use.

The BaseImageTransformer subclass is responsible for transforming a variable-sized image to a fixed-sized one through some sort of preprocessing and is used by the dataset to construct batches of fixed-sized examples.

There is currently only one subclass implemented, ift6266h15.code.pylearn2.dataset.variable_image_dataset.RandomCrop, which scales the input image so that its smallest side has length scaled_size, and takes a random square crop of dimension crop_size inside the scaled image (both scaled_size and crop_size are constructor arguments).

Here’s how we would instantiate and iterate over the dataset:

from ift6266h15.code.pylearn2.datasets.variable_image_dataset import DogsVsCats, RandomCrop
dataset = DogsVsCats(
    RandomCrop(256, 221),
    start=0, stop=20000)
iterator = dataset.iterator(
    mode='batchwise_shuffled_sequential',
    batch_size=100)
for X, y in iterator:
    print X.shape, y.shape

Note that by default the dataset iterates over both features and targets.

Here’s how you would use the dataset inside a YAML file to train a linear classifier on the dataset:

!obj:pylearn2.train.Train {
    dataset: &train !obj:ift6266h15.code.pylearn2.datasets.variable_image_dataset.DogsVsCats {
        transformer: &transformer !obj:ift6266h15.code.pylearn2.datasets.variable_image_dataset.RandomCrop {
            scaled_size: 256,
            crop_size: 221,
        },
        start: 0,
        stop: 20000,
    },
    model: !obj:pylearn2.models.mlp.MLP {
        nvis: 146523,
        layers: [
            !obj:pylearn2.models.mlp.Softmax {
                layer_name: 'y',
                n_classes: 2,
                irange: 0.01,
            },
        ],
    },
    algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {
        batch_size: &batch_size 100,
        train_iteration_mode: 'batchwise_shuffled_sequential',
        batches_per_iter: 10,
        monitoring_batch_size: *batch_size,
        monitoring_batches: 10,
        monitor_iteration_mode: 'batchwise_shuffled_sequential',
        learning_rate: 1e-3,
        learning_rule: !obj:pylearn2.training_algorithms.learning_rule.Momentum {
            init_momentum: 0.95
        },
        monitoring_dataset: {
            'train' : *train,
            'valid': !obj:ift6266h15.code.pylearn2.datasets.variable_image_dataset.DogsVsCats {
                transformer: *transformer,
                start: 20000,
                stop: 25000,
            },
        },
        cost: !obj:pylearn2.costs.cost.MethodCost {
            method: 'cost_from_X',
        },
        termination_criterion: !obj:pylearn2.termination_criteria.EpochCounter {
            max_epochs: 10
        },
    },
}

This YAML file trains a softmax classifier for 10 epochs using the first 20,000 examples of the training set. An epoch consists of 10 batches of 100 random examples. Monitoring values are approximated with 10 batches of 100 random examples.

Implementing a BaseImageTransformer subclass

In order to implement your own preprocessing of the images, your BaseImageTransformer subclass needs to implement two methods: get_shape and preprocess.

The get_shape method needs to return the width and height of preprocessed images.

The preprocess method does the actual preprocessing. Given an input image, it returns the preprocessed version whose shape needs to be consistent with what get_shape returns.

Have a look at how RandomCrop is implemented to get a better feel of how it’s done.

Clarifications for Jan. 22’s lecture

Theano port

As requested by a student today, the solution to last week’s assignment was updated to include the Theano port that was made in class.

Regarding MethodCost

The MethodCost subclass of Cost has a constructor which accepts a string representing the method to call on the model to get a cost expression.

A coupling between the Cost subclass and the Model subclass was mentioned today: the Cost subclass is responsible to know what method of the Model subclass in order to produce a cost expression from the inputs. This method can either

  1. produce a prediction, in which case the Cost subclass needs to combine this prediction with the targets to form the cost expression (this was the case for the MLP implementation presented today), or
  2. produce the cost expression directly from the inputs and the targets, in which case the Cost subclass only acts as an intermediary.

This second option is how MethodCost is implemented: when its expr method is called, it calls whatever method name was passed as argument to its constructor on the Model subclass by giving it the inputs and the targets and returns the returned value as the cost expression.

In the case of the MLP class (the one in Pylearn2, not the small implementation presented today), the cost_from_X method takes inputs and targets and produces a cost expression based on its output layer, which is why the cost was specified as !obj:pylearn2.costs.cost.MethodCost {method: ‘cost_from_X’} in the example YAML file.

Preparing for the Pylearn2 primer

In preparation for the lecture given on Thursday, January 22, 2015, those who would like to follow along should install Pylearn2 on their machine. Complete installation instructions can be found on Pylearn2’s online documentation.

We will cover how to install Pylearn2 on a machine which already meets the requirements. For information on how to install prerequisites, refer to your platform-specific documentation.

Requirements

  • Theano (bleeding-edge installation) and its dependencies
  • PyYAML
  • PIL (for some image-related functionalities
  • setuptools or distribute

Installation instructions

These instructions will work on a Linux or Mac OS machine when typed in a bash shell. You may need to adapt the commands to your environment.

  1. Change directory to where you’d like the Pylearn2 codebase to reside (the directory you chose for your Theano installation is not a bad idea).
  2. Run
    git clone git://github.com/lisa-lab/pylearn2.git
  3. Change directory inside the Pylearn2 directory and type
    python setup.py develop --user

    This will ensure that the Pylearn2 repository you just checked out will be found by the Python interpreter. Every change in the Pylearn2 repository is now automatically present when you import the pylearn2 module without you needing to reinstall anything. The command “installs” Pylearn2 locally; if you’d prefer to have a global installation, remove the –user flag (some sudo’ing legwork may be required in that case).

  4. To confirm that everything works fine, make sure you’re outside of the Pylearn2 directory, launch a Python interpreter and type
    import pylearn2

    If you see no error, the installation was successful!

Keeping things up to date

Pylearn2 is regularly updated. In order to get the latest changes, simply run

git pull origin master

inside the Pylearn2 directory.

Preparing for Thursday’s lecture

In preparation for the lecture given on Thursday, January 15, 2015, those who would like to follow along should install Theano on their machine. Complete installation instructions can be found on Theano’s online documentation.

We will cover how to install Theano on a machine which already meets the requirements. For information on how to install prerequisites, refer to your platform-specific documentation.

Because we intend to install Pylearn2 later on, we will be performing a bleeding-edge installation for the Github repository.

Requirements

  • Git
  • Linux, Mac OS X or Windows operating system (Note that Linux is the most tested platform.)
  • Python 2.x >= 2.6 (The development package (python-dev or python-devel on most Linux distributions) is recommended.)
  • g++, python-dev (Not technically required but highly recommended, in order to compile generated C code. Theano can fall back on a NumPy-based Python execution model, but a C compiler allows for vastly faster execution. g++ >= 4.2 (for openmp that is currently always used) more recent version recommended!)
  • setuptools or distribute
  • numpy >= 1.5.0 (Earlier versions could work, but we don’t test it.)
  • scipy (Only currently required for sparse matrix and special functions support, but highly recommended. We recommend SciPy >=0.8 if you are using sparse matrices)
  • A BLAS installation with Level 3 functionality (Including the development headers (-dev, -devel, depending on your Linux distribution).N.B.: OSX users already have a BLAS installation through the Accelerate framework.

Installation instructions

These instructions will work on a Linux or Mac OS machine when typed in a bash shell. You may need to adapt the commands to your environment.

  1. Change directory to where you’d like the Theano codebase to reside (e.g. in your home directory, or in some place where you usually put your code).
  2. Run
    git clone git://github.com/Theano/Theano.git
  3. Change directory inside the Theano directory and type
    python setup.py develop --user

    This will ensure that the Theano repository you just checked out will be found by the Python interpreter. Every change in the Theano repository is now automatically present when you import the theano module without you needing to reinstall anything. The command “installs” Theano locally; if you’d prefer to have a global installation, remove the –user flag (some sudo’ing legwork may be required in that case).

  4. To confirm that everything works fine, make sure you’re outside of the Theano directory, launch a Python interpreter and type
    import theano

    If you see no error, the installation was successful!

Keeping things up to date

Theano is regularly updated. In order to get the latest changes, simply run

git pull origin master

inside the Theano directory.