Import a model

Importing a model from ONNX

nGraph APIs can be used to run inference on a model that has been exported from a Deep Learning framework. An export produces a file with a serialized model that can be loaded and passed to one of the nGraph backends.

Importing a model from ONNX

The most-widely supported export format available today is ONNX. Models that have been serialized to ONNX are easy to identify; they are usually named <some_model>.onnx or <some_model>.onnx.pb. These tutorials from ONNX describe how to turn trained models into an .onnx export.

Important

If you landed on this page and you already have an .onnx or .onnx.pb formatted file, you should be able to run the inference without needing to dig into anything from the “Frameworks” sections. You will, however, need to have completed the steps outlined in our Install guide.

To demonstrate functionality, we’ll use an already serialized CIFAR10 model trained via ResNet20. Remember that this model has already been trained and exported from a framework such as Caffe2, PyTorch or CNTK; we are simply going to build an nGraph representation of the model, execute it, and produce some outputs.

Installing ngraph_onnx with nGraph from scratch

To use ONNX models with nGraph, you will also need the companion tool ngraph_onnx, which requires Python 3.4 or higher. If nGraph has not yet been installed to your system, you can follow these steps to install everything at once; if an ngraph_dist is already installed on your system, skip ahead to the next section, Installing ngraph-onnx.

  1. Install prerequisites for the system and install nGraph as ngraph_dist:

    $ apt update
    $ apt install python3 python3-pip python3-dev python3-venv
    $ apt install build-essential cmake curl clang-3.9 git zlib1g zlib1g-dev libtinfo-dev
    $ git clone https://github.com/NervanaSystems/ngraph.git
    $ cd ngraph && mkdir build
    $ cd build && cmake ../ -DNGRAPH_USE_PREBUILT_LLVM=TRUE
    $ make install
    
  2. Build the Python package (binary wheel) for ngraph and set up an env for ONNX:

    $ cd ngraph/python
    $ git clone --recursive -b allow-nonconstructible-holders https://github.com/jagerman/pybind11.git
    $ export PYBIND_HEADERS_PATH=$PWD/pybind11
    $ export NGRAPH_CPP_BUILD_PATH=$HOME/ngraph_dist
    $ python3 setup.py bdist_wheel
    $ cd .. python3 -m venv onnx
    $ cd onnx/
    $ . bin/activate
    
  3. Check for the binary wheel file under /ngraph/python/dist/ and install it with pip.

    (onnx)$ pip install -U python/dist/ngraph-0.2.0-cp35-cp35m-linux_x86_64.whl
    
  4. Confirm ngraph is properly installed through a Python interpreter:

    (onnx)$ python3
    
    import ngraph as ng
    ng.abs([[1, 2, 3], [4, 5, 6]])
    <Abs: 'Abs_1' ([2, 3])>
    

    If you don’t see any errors, ngraph should be installed correctly.

Installing ngraph-onnx

Add the dependencies for ONNX:

$ apt install protobuf-compiler libprotobuf-dev

Install the ngraph-onnx companion tool using pip:

(onnx) $ pip install git+https://github.com/NervanaSystems/ngraph-onnx/

Importing a serialized model

With the dependencies added, we can now import a model that has been serialized by ONNX, interact locally with the model by running Python code, create and load objects, and run inference.

This section assumes that you have your own ONNX model. With this example model from Microsoft*’s Deep Learning framework, CNTK, we can outline the procedure to show how to run ResNet on model that has been trained on the CIFAR10 data set and serialized with ONNX.

(Optional) Localize your export to the virtual environment

For this example, let’s say that our serialized file was output under our $HOME directory, say at ~/onnx_conversions/trained_model.onnx. To make loading this file easier, you can run the example below from your Venv in that directory. If you invoke your python interpreter in a different directory, you will need to specify the relative path to the location of the .onnx file.

Important

If you invoke your Python interpreter in directory other than where you outputted your trained model, you will need to specify the relative path to the location of the .onnx file.

(onnx) $ cd ~/onnx_conversions
(onnx) $ python3

Enable ONNX and load an ONNX file from disk

import onnx

onnx_protobuf = onnx.load('/path/to/model/cntk_ResNet20_CIFAR10/model.onnx')

Convert an ONNX model to an ngraph model

from ngraph_onnx.onnx_importer.importer import import_onnx_model
ng_model = import_onnx_model(onnx_protobuf)[0]

The importer returns a list of ngraph models for every ONNX graph output:

print(ng_models)
[{
    'name': 'Plus5475_Output_0',
    'output': <Add: 'Add_1972' ([1, 10])>,
    'inputs': [<Parameter: 'Parameter_1104' ([1, 3, 32, 32], float)>]
 }]

The output field contains the ngraph node corrsponding to the output node in the imported ONNX computational graph. The inputs list contains all input parameters for the computation which generates the output.

Using ngraph_api, create a callable computation object

import ngraph as ng
runtime = ng.runtime(backend_name='CPU')
resnet = runtime.computation(ng_model['output'], *ng_model['inputs'])

Load or create an image

import numpy as np
picture = np.ones([1, 3, 32, 32])

Run ResNet inference on picture

resnet(picture)

Put it all together

“Demo sample code to run inference with nGraph”
import onnx

onnx_protobuf = onnx.load('/path/to/model/cntk_ResNet20_CIFAR10/model.onnx')

# Convert a serialized ONNX model to an ngraph model
from ngraph_onnx.onnx_importer.importer import import_onnx_model
ng_model = import_onnx_model(onnx_protobuf)[0]


# Using an ngraph runtime (CPU backend), create a callable computation
import ngraph as ng
runtime = ng.runtime(backend_name='CPU')
resnet = runtime.computation(ng_model['output'], *ng_model['inputs'])

# Load or create an image
import numpy as np
picture = np.ones([1, 3, 32, 32])

# Run ResNet inference on picture
resnet(picture)

Outputs will vary greatly, depending on your model; for demonstration purposes, the code will look something like:

array([[ 1.312082 , -1.6729496,  4.2079577,  1.4012241, -3.5463796,
      2.3433776,  1.7799224, -1.6155214,  0.0777044, -4.2944093]],
   dtype=float32)