In this phase of my experiments I’ve tested the following preprocessing algorithms in the pylearn2:
1) Standardization: This centers the columns of the image matrix by subtracting the mean and reduces the variance of the data by dividing by standard deviation.
2) Global Contrast Normalization: This class computes the mean and standard deviation of the all pixels in the image and subtracts the mean from each pixel and divides the pixels by the standard deviation.
3) Whitening: This is ZCA whitening is used for decorrelating the pixels in the image and removing the redundancy in the images. Whitening is one of the most used preprocessing tool in deep learning. It usually improves the results.
4) Standardization + Whitening: It is known that Whitening and PCA works better if your data is mean centered. Standardization centers the data and therefore I expect that Standardized+Whitened Images will perform better.
5) Standardization + GCN: I have added this one, just to see if it will improve the results.
Analysing the dataset
Before starting working on a dataset I find it crucial to do at least some basic statistical analysis about the properties of the dataset which may be helpful to have some basic idea about the dataset.
Statistical properties
-First order statistics
Mean: 124.525897579
Standard deviation: 50.7928015714
-Higher order statistics:
Kurtosis:-0.256715476375
Skewness: 0.212287734117
Histogram of the dataset
While I was working with pylearn2’s pipeline interface, I have realized a few small bugs in there and fixed them: https://github.com/lisa-lab/pylearn2/commit/b4d74eff8d492bf7f31cfee9a669d7b5ab9c3ad4
Apart from these informations images are 48×48 and ubyte single channel i.e. grayscale and intensities are between 0 and 255.
In my evaluation I’ve used a Rectifier nonlinearity with two layers containing 4000 hidden units at each layer with decaying learning rate and momentum.
As seen from the pixel intensity values in the histogram and the kurtosis and skewness values, the pixels are distributed almost according to a gaussian distribution. This is interesting because usually natural images are assumed to have a heavy-tailed distribution.
Training Error | Test Error | |
Standardization | 0.062 | 0.219 |
GCN | 0.021 | 0.272 |
ZCA Whitening | 0.000 | 0.337 |
Standardization + Whitening | 0.000 | 0.292 |
Standardization + GCN | 0.062 | 0.219 |
As seen from the table ZCA leads to terrible results. I think this is due to the gaussian like distribution of the dataset. Because ZCA and PCA whitening works best if your pixel intensities are distributed according to a heavy tailed distribution such as laplacian or student’s t distribution.
Here are some weights learned with different preprocessing: