Interact with Backends

Backend

Backends are responsible for function execution and value allocation. They can be used to carry out a programmed computation from a framework by using a CPU or GPU; or they can be used with an Interpreter mode, which is primarily intended for testing, to analyze a program, or for a framework developer to develop a custom UI or API.

../_images/runtime.png
class Backend

Interface to a generic backend.

Backends are responsible for function execution and value allocation.

Subclassed by ngraph::runtime::cpu::CPU_Backend, ngraph::runtime::gpu::GPU_Backend, ngraph::runtime::intelgpu::IntelGPUBackend, ngraph::runtime::interpreter::INTBackend

Public Functions

virtual std::shared_ptr<ngraph::runtime::TensorView> create_tensor(const ngraph::element::Type &element_type, const Shape &shape) = 0

Create a tensor specific to this backend.

Return
shared_ptr to a new backend specific tensor
Parameters
  • element_type: The type of the tensor element
  • shape: The shape of the tensor

virtual std::shared_ptr<ngraph::runtime::TensorView> create_tensor(const ngraph::element::Type &element_type, const Shape &shape, void *memory_pointer) = 0

Create a tensor specific to this backend.

Return
shared_ptr to a new backend specific tensor
Parameters
  • element_type: The type of the tensor element
  • shape: The shape of the tensor
  • memory_pointer: A pointer to a buffer used for this tensor. The size of the buffer must be sufficient to contain the tensor. The lifetime of the buffer is the responsibility of the caller.

template <typename T>
std::shared_ptr<ngraph::runtime::TensorView> create_tensor(const Shape &shape)

Create a tensor of C type T specific to this backend.

Return
shared_ptr to a new backend specific tensor
Parameters
  • shape: The shape of the tensor

virtual bool compile(std::shared_ptr<Function> func) = 0

Compiles a Function.

Return
true if compile is successful, false otherwise
Parameters
  • func: The function to compile

virtual bool call(std::shared_ptr<Function> func, const std::vector<std::shared_ptr<runtime::TensorView>> &outputs, const std::vector<std::shared_ptr<runtime::TensorView>> &inputs) = 0

Executes a single iteration of a Function. If func is not compiled the call will compile it.

Return
true if iteration is successful, false otherwise
Parameters
  • func: The function to execute

bool call_with_validate(std::shared_ptr<Function> func, const std::vector<std::shared_ptr<runtime::TensorView>> &outputs, const std::vector<std::shared_ptr<runtime::TensorView>> &inputs)

Executes a single iteration of a Function. If func is not compiled the call will compile it. Optionally validates the inputs and outputs against the function graph.

Return
true if iteration is successful, false otherwise
Parameters
  • func: The function to execute

void remove_compiled_function(std::shared_ptr<Function> func)

Compiled functions may be cached. This function removes a compiled function from the cache.

Parameters
  • func: The function to execute

virtual void enable_performance_data(std::shared_ptr<Function> func, bool enable)

Enable the collection of per op performance information on a specified Function. Data is collection via the get_performance_data method.

Parameters
  • func: The function to collect perfomance data on.
  • enable: Set to true to enable or false to disable data collection

vector<ngraph::runtime::PerformanceCounter> get_performance_data(std::shared_ptr<Function> func) const

Collect performance information gathered on a Function.

Return
Vector of PerformanceCounter information.
Parameters
  • func: The function to get collected data.

Public Static Functions

shared_ptr<runtime::Backend> create(const std::string &type)

Create a new Backend object.

Return
shared_ptr to a new Backend or nullptr if the named backend does not exist.
Parameters
  • type: The name of a registered backend, such as “CPU” or “GPU”. To select a subdevice use “GPU:N” where sN is the subdevice number.

vector<string> get_registered_devices()

Query the list of registered devices.

Return
A vector of all registered devices.

TensorView

class TensorView

Subclassed by ngraph::runtime::cpu::CPUTensorView, ngraph::runtime::gpu::GPU_TensorView, ngraph::runtime::HostTensorView, ngraph::runtime::intelgpu::IntelGPUTensorView

Public Functions

virtual void write(const void *p, size_t tensor_offset, size_t n) = 0

Write bytes directly into the tensor.

Parameters
  • p: Pointer to source of data
  • tensor_offset: Offset into tensor storage to begin writing. Must be element-aligned.
  • n: Number of bytes to write, must be integral number of elements.

virtual void read(void *p, size_t tensor_offset, size_t n) const = 0

Read bytes directly from the tensor.

Parameters
  • p: Pointer to destination for data
  • tensor_offset: Offset into tensor storage to begin reading. Must be element-aligned.
  • n: Number of bytes to read, must be integral number of elements.