Distributed training with nGraph

Important

Distributed training is not officially supported as of version 0.18; however, some configuration options have worked for nGraph devices in testing environments.

How? (Generic frameworks)

To synchronize gradients across all workers, the essential operation for data parallel training, due to its simplicity and scalability over parameter servers, is allreduce. The AllReduce op is one of the nGraph Library’s core ops. To enable gradient synchronization for a network, we simply inject the AllReduce op into the computation graph, connecting the graph for the autodiff computation and optimizer update (which then becomes part of the nGraph graph). The nGraph Backend will handle the rest.

Data scientists with locally-scalable rack or cloud-based resources will likely find it worthwhile to experiment with different modes or variations of distributed training. Deployments using nGraph Library with supported backends can be configured to train with data parallelism and will soon work with model parallelism. Distributing workloads is increasingly important, as more data and bigger models mean the ability to Distribute training across multiple nGraph backends work with larger and larger datasets, or to work with models having many layers that aren’t designed to fit to a single device.

Distributed training with data parallelism splits the data and each worker node has the same model; during each iteration, the gradients are aggregated across all workers with an op that performs “allreduce”, and applied to update the weights.

Using multiple machines helps to scale and speed up deep learning. With large mini-batch training, one could train ResNet-50 with Imagenet-1k data to the Top 5 classifier in minutes using thousands of CPU nodes. See arxiv.org/abs/1709.05011.

Future work

More communication ops support is in the works. See also: List of passes.