One method for increasing the accuracy of a learning system is to train multiple instances and gather them into an ensemble. During classification, each instance is polled, and the average of the votes determines the overall output. For this project I created a simple neural network ensemble that's designed to train and classify the MNIST handwritten digits set. This is a relatively simple approach, yet it significantly outperforms my randomized decision forest implementation in terms of training speed, memory footprint, and accuracy.


 Approach Training speed Memory footprint Accuracy
 Neural Network Ensemble <30 minutes 12 MB 96.75%
 Randomized Decision Forest >24 hours 40 MB 96.5%


Features

  • Configuration: control epoch size, learning rate, mini-batch size, and layer dimensions (size and types).

  • Tuning: quadratic and cross-entropy cost, L2 regularization, multiple activation functions (tanh, relu, sigmoid).

  • Serialization: save and load network ensembles for quick reuse.



Downloads

  Neural Network Ensemble Demo for Windows (32 KB)
  MNIST Training and Test Samples (10.5 MB)
  Neural Network Ensemble (96.75% accuracy) (12 MB)
  Neural Network Ensemble Source Code (C++)



Demo Instructions

If you'd like to test out the pre-built app, or use the source as-is, then you'll need to download the MNIST training and test data and unzip it into the same folder as the demo program. Next, you could train your own ensemble, but I'd recommend that you save yourself some time and use my existing 3 network ensemble by unzipping it to the same folder as the program.

Running the program (on Windows) without any parameters will display the prompt below. You train one network at a time and then combine them into an ensemble that you can use for verification. If you'd like to modify any of the hyper-parameters then you'll need to download, modify, and build the source code. 


Usage: simple-nn-x64.exe [options]
  --train  [output network filename]                            Trains a single neural network based on the MNIST dataset.
  --combine [output ensemble filename] [network filenames]      Combines a set of network files into a single ensemble file.
  --verify [input ensemble filename]                            Tests the accuracy of a neural network ensemble against the MNIST test set.