neon™

The neon™ frontend to Intel® Nervana™ graph (ngraph) provides common deep learning primitives, such as activation functions, optimizers, layers, and more. We include several examples in this release to illustrate how to use the neon frontend to construct your models:

  • examples/minst/mnist_mlp.py: Multi-layer perceptron on the MNIST digits dataset.
  • examples/cifar10/cifar10_mlp.py: Multi-layer perceptron on the CIFAR10 dataset.
  • examples/cifar10/cifar10_conv.py: Convolutional neural networks applied to the CIFAR10 dataset.
  • examples/ptb/char_rnn.py: Character-level RNN language model on the Penn Treebank dataset.

We currently have support for the following sets of deep learning primitives:

  • Layers: Linear, Bias, Conv2D, Pool2D, BatchNorm, Dropout, Recurrent
  • Activations: Rectlin, Rectlinclip, Identity, Explin, Normalizer, Softmax, Tanh, Logistic
  • Initializers: GaussianInit, UniformInit, ConstantInit
  • Optimizers: GradientDescentMomentum, RMSprop
  • Callbacks: TrainCostCallback, RunTimerCallback, ProgressCallback, TrainLoggerCallback, LossCallback

MNIST Example

The mnist_mlp.py example provides an introduction into the neon frontend. Similiar to neon, we begin with an arg parser that allows command line flags for specifying options, such as batch size and the data directory:

from ngraph.frontends.neon import NgraphArgparser
parser = NgraphArgparser(description='Train simple mlp on mnist dataset')
args = parser.parse_args()

To provision data to the model, we use the ArrayIterator object, which is a Python iterator that returns a minibatch of inputs and targets to the model with each call. We also provide a helper function for handling the MNIST dataset and providing NumPY arrays with the image and target data.

from mnist import MNIST
from ngraph.frontends.neon import ArrayIterator

# Create the dataloader
train_data, valid_data = MNIST(args.data_dir).load_data()
train_set = ArrayIterator(train_data, args.batch_size, total_iterations=args.num_iterations)
valid_set = ArrayIterator(valid_data, args.batch_size)

You can compose models as a list of layers which then gets passed to a container that handles how the layers are linked. Here we just have a sequential list of layers, so we use the Sequential container.

from ngraph.frontends.neon import Affine, Preprocess, Sequential
from ngraph.frontends.neon import GaussianInit, Rectlin, Logistic, GradientDescentMomentum

seq1 = Sequential([Preprocess(functor=lambda x: x / 255.),
                   Affine(nout=100, weight_init=GaussianInit(), activation=Rectlin()),
                   Affine(axes=ax.Y, weight_init=GaussianInit(), activation=Logistic())])

Note that above we also use a Preprocess layer to scale the input image data to between 0 and 1.

The neon frontend has a predefined set of axes commonly used with deep learning in the ngraph/frontends/neon/axis.py file. We import the name_scope from this file as ax, and can define the lengths of the relevant axes give the shape of the input image and the target number of classes:

from ngraph.frontends.neon import ax

ax.C.length, ax.H.length, ax.W.length = train_set.shapes['image']
ax.N.length = args.batch_size
ax.Y.length = 10

With these axes defined, we can then define placeholders for the inputs. Our image has axes CHWN, and the target label has axes N for the batch size.

inputs = dict(image=ng.placeholder([ax.C, ax.H, ax.W, ax.N]),
              label=ng.placeholder([ax.N]))

We use stochastic gradient descent with momentum.

optimizer = GradientDescentMomentum(0.1, 0.9)

We then define the model output, and the associated cost function and metric (the misclassification rate) using the ngraph API directly:

output_prob = seq1.train_outputs(inputs['image'])

errors = ng.not_equal(ng.argmax(output_prob, out_axes=[ax.N]), inputs['label'])
loss = ng.cross_entropy_binary(output_prob, ng.one_hot(inputs['label'], axis=ax.Y))

mean_cost = ng.mean(loss, out_axes=())
updates = optimizer(loss)

To obtain the model output, we use the sequential container’s included train_outputs() method, which essentially performs the forward pass through the layers of the model.

Now that we have used the neon frontend to compose our graph, we pass it to a transformer for execution by specifying the computations required to both train the network and also to compute the loss. Instead of directly specifying the computations using transformer.computation() as with the Intel Nervana graph walkthrough examples, we instead use a helper function make_bound_computation() to create computations that bind a set of inputs with outputs. We can specify a set of outputs using Python dictionaries.

from ngraph.frontends.neon import make_bound_computation, make_default_callbacks

train_outputs = dict(batch_cost=mean_cost, updates=updates)
loss_outputs = dict(cross_ent_loss=loss, misclass_pct=errors)

# Now bind the computations we are interested in
transformer = ngt.make_transformer()
train_computation = make_bound_computation(transformer, train_outputs, inputs)
loss_computation = make_bound_computation(transformer, loss_outputs, inputs)

In the case of train_computation, we can think of make_bound_computation as creating a computation by calling transformer.computation([mean_cost updates], inputs).

Callbacks allow the model to report back its progress and any relevant metrics during the course of training.

from ngraph.frontends.neon import make_default_callbacks

cbs = make_default_callbacks(transformer=transformer,
                         output_file=args.output_file,
                     frequency=args.iter_interval,
                     train_computation=train_computation,
                     total_iterations=args.num_iterations,
                     eval_set=valid_set,
                     loss_computation=loss_computation,
                     use_progress_bar=args.progress_bar)

Finally, we use another helper function, loop_train, to train the model. loop_train loops through the provided training data, calling the provided computation (in this case train_computation), to update the model weights and report progress via the provided callbacks.

from ngraph.frontends.neon import loop_train

loop_train(train_set, train_computation, cbs)

Note

This model is very similar to the MNIST_Direct.ipynb, which walks through an implementation using the Intel Nervana graph API directly instead of the neon frontend. The neon frontend essentially contains objects and helper methods that wrap the ngraph calls to make it easier for users to compose the networks in terms of deep learning building blocks.