Backend APIs

As of version 0.15, there is a new backend API to work with functions that can be compiled as a runtime Executable. Where previously Backend used a shared_ptr<Function> as the handle passed to the call method to execute a compiled object, the addition of the shared_ptr<Executable> object has more direct methods to actions such as validate, call, get_performance_data, and so on. This new API permits any executable to be saved or loaded into or out of storage and makes it easier to distinguish when a Function is compiled, thus making the internals of the Backend and Executable easier to implement.

How to use?

  1. Create a Backend; think of it as a compiler.
  2. A Backend can then produce an Executable by calling compile.
  3. A single iteration of the executable is executed by calling the call method on the Executable object.

Backend

../_images/backend-dgm.png

Various backends are accessible via nGraph core APIs

class Backend

Interface to a generic backend.

Backends are responsible for function execution and value allocation.

Subclassed by ngraph::runtime::cpu::CPU_Backend, ngraph::runtime::gcpu::GCPUBackend, ngraph::runtime::gpu::GPU_Backend, ngraph::runtime::hybrid::HybridBackend, ngraph::runtime::intelgpu::IntelGPUBackend, ngraph::runtime::interpreter::INTBackend, ngraph::runtime::nop::NOPBackend, ngraph::runtime::plaidml::PlaidML_Backend

Public Types

enum Property

A set of properties supported by a backend.

Values:

memory_attach

New tensor can use attached memory.

Public Functions

virtual std::shared_ptr<ngraph::runtime::Tensor> create_tensor(const ngraph::element::Type &element_type, const Shape &shape) = 0

Create a tensor specific to this backend.

Return
shared_ptr to a new backend-specific tensor
Parameters
  • element_type: The type of the tensor element
  • shape: The shape of the tensor

virtual std::shared_ptr<ngraph::runtime::Tensor> create_tensor(const ngraph::element::Type &element_type, const Shape &shape, void *memory_pointer) = 0

Create a tensor specific to this backend.

Return
shared_ptr to a new backend-specific tensor
Parameters
  • element_type: The type of the tensor element
  • shape: The shape of the tensor
  • memory_pointer: A pointer to a buffer used for this tensor. The size of the buffer must be sufficient to contain the tensor. The lifetime of the buffer is the responsibility of the caller.

template <typename T>
std::shared_ptr<ngraph::runtime::Tensor> create_tensor(const Shape &shape)

Create a tensor of C type T specific to this backend.

Return
shared_ptr to a new backend specific tensor
Parameters
  • shape: The shape of the tensor

virtual std::shared_ptr<Executable> compile(std::shared_ptr<Function> func, bool enable_performance_data = false) = 0

Compiles a Function.

Return
compiled function or nullptr on failure
Parameters
  • func: The function to compile

bool is_supported(const Node &node) const

Test if a backend is capable of supporting an op.

Return
true if the op is supported, false otherwise.
Parameters
  • node: is the op to test.

bool is_supported_property(const Property prop) const

Test if a backend particular property is supported.

Return
true if the property is supported, false otherwise.
Parameters
  • prop: is the feature to test.

bool call_with_validate(std::shared_ptr<Executable> handle, const std::vector<std::shared_ptr<runtime::Tensor>> &outputs, const std::vector<std::shared_ptr<runtime::Tensor>> &inputs)

The following methods are temporary hacks to reduce the number of changes in this PR They will be removed in a follow-on PR

Public Static Functions

unique_ptr<runtime::Backend> create(const std::string &type)

Create a new Backend object.

Return
unique_ptr to a new Backend or nullptr if the named backend does not exist.
Parameters
  • type: The name of a registered backend, such as “CPU” or “GPU”. To select a subdevice use “GPU:N” where s`N` is the subdevice number.

vector<string> get_registered_devices()

Query the list of registered devices.

Return
A vector of all registered devices.

Executable

../_images/runtime-exec.png

The compile function on an Executable has more direct methods to actions such as validate, call, get_performance_data, and so on.

class Executable

Subclassed by ngraph::runtime::cpu::CPU_Executable, ngraph::runtime::gcpu::GCPUExecutable, ngraph::runtime::gpu::GPU_Executable, ngraph::runtime::hybrid::HybridExecutable, ngraph::runtime::intelgpu::IntelGPUExecutable, ngraph::runtime::interpreter::INTExecutable, ngraph::runtime::nop::NOPExecutable, ngraph::runtime::plaidml::PlaidML_Executable

Public Functions

virtual bool call(const std::vector<std::shared_ptr<runtime::Tensor>> &outputs, const std::vector<std::shared_ptr<runtime::Tensor>> &inputs) = 0

Return
true if iteration is successful, false otherwise
Parameters

bool call_with_validate(const std::vector<std::shared_ptr<runtime::Tensor>> &outputs, const std::vector<std::shared_ptr<runtime::Tensor>> &inputs)

Executes a single iteration of a Function.

Return
true if iteration is successful, false otherwise
Parameters

vector<runtime::PerformanceCounter> get_performance_data() const

Collect performance information gathered on a Function.

Return
Vector of PerformanceCounter information.

void validate(const std::vector<std::shared_ptr<runtime::Tensor>> &outputs, const std::vector<std::shared_ptr<runtime::Tensor>> &inputs)

Validates a Function.

Parameters

const ngraph::ParameterVector &get_parameters() const

Query the input Parameters.

Return
an ngraph::op::ParameterVector of all input parameters

const ngraph::ResultVector &get_results() const

Query the output Results.

Return
an ngraph::ResultVector of all input parameters

Tensor

class Tensor

Subclassed by ngraph::runtime::cpu::CPUTensorView, ngraph::runtime::gpu::GPUTensor, ngraph::runtime::HostTensor, ngraph::runtime::intelgpu::IntelGPUTensorView, ngraph::runtime::plaidml::PlaidML_Tensor

Public Functions

const Shape &get_shape() const

Get tensor shape.

Return
const reference to a Shape

Strides get_strides() const

Get tensor strides.

Return
Strides

const element::Type &get_element_type() const

Get tensor element type.

Return
element::Type

size_t get_element_count() const

Get number of elements in the tensor.

Return
number of elements in the tensor

size_t get_size_in_bytes() const

Get the size in bytes of the tensor.

Return
number of bytes in tensor’s allocation

const std::string &get_name() const

Get tensor’s unique name.

Return
tensor’s name

shared_ptr<descriptor::layout::TensorLayout> get_tensor_layout() const

Get tensor layout.

Return
tensor layout

void set_tensor_layout(const std::shared_ptr<descriptor::layout::TensorLayout> &layout)

Set tensor layout.

Parameters
  • layout: Layout to set

bool get_stale() const

Get the stale value of the tensor. A tensor is stale if its data is changed.

Return
true if there is new data in this tensor

void set_stale(bool val)

Set the stale value of the tensor. A tensor is stale if its data is changed.

virtual void write(const void *p, size_t offset, size_t n) = 0

Write bytes directly into the tensor.

Parameters
  • p: Pointer to source of data
  • offset: Offset into tensor storage to begin writing. Must be element-aligned.
  • n: Number of bytes to write, must be integral number of elements.

virtual void read(void *p, size_t offset, size_t n) const = 0

Read bytes directly from the tensor.

Parameters
  • p: Pointer to destination for data
  • offset: Offset into tensor storage to begin writing. Must be element-aligned.
  • n: Number of bytes to read, must be integral number of elements.

void copy_from(const ngraph::runtime::Tensor &source)

copy bytes directly from source to this tensor

Parameters
  • source: The source tensor

HostTensor

class HostTensor : public ngraph::runtime::Tensor

Public Functions

void write(const void *p, size_t tensor_offset, size_t n)

Write bytes directly into the tensor.

Parameters
  • p: Pointer to source of data
  • tensor_offset: Offset into tensor storage to begin writing. Must be element-aligned.
  • n: Number of bytes to write, must be integral number of elements.

void read(void *p, size_t tensor_offset, size_t n) const

Read bytes directly from the tensor.

Parameters
  • p: Pointer to destination for data
  • tensor_offset: Offset into tensor storage to begin reading. Must be element-aligned.
  • n: Number of bytes to read, must be integral number of elements.

PlaidML

class PlaidML_Backend : public ngraph::runtime::Backend

Public Functions

std::shared_ptr<ngraph::runtime::Tensor> create_tensor(const ngraph::element::Type &element_type, const Shape &shape)

Create a tensor specific to this backend.

Return
shared_ptr to a new backend-specific tensor
Parameters
  • element_type: The type of the tensor element
  • shape: The shape of the tensor

std::shared_ptr<ngraph::runtime::Tensor> create_tensor(const ngraph::element::Type &element_type, const Shape &shape, void *memory_pointer)

Create a tensor specific to this backend.

Return
shared_ptr to a new backend-specific tensor
Parameters
  • element_type: The type of the tensor element
  • shape: The shape of the tensor
  • memory_pointer: A pointer to a buffer used for this tensor. The size of the buffer must be sufficient to contain the tensor. The lifetime of the buffer is the responsibility of the caller.

std::shared_ptr<ngraph::runtime::Executable> compile(std::shared_ptr<Function> func, bool enable_performance_data = false)

Compiles a Function.

Return
compiled function or nullptr on failure
Parameters
  • func: The function to compile

bool is_supported(const Node &node) const

Test if a backend is capable of supporting an op.

Return
true if the op is supported, false otherwise.
Parameters
  • node: is the op to test.

bool is_supported_property(const Property prop) const

Test if a backend particular property is supported.

Return
true if the property is supported, false otherwise.
Parameters
  • prop: is the feature to test.