TfLiteContext

#include <common.h>

TfLiteContext allows an op to access the tensors.

Summary

TfLiteContext is a struct that is created by the TF Lite runtime and passed to the "methods" (C function pointers) in the TfLiteRegistration struct that are used to define custom ops and custom delegate kernels. It contains information and methods (C function pointers) that can be called by the code implementing a custom op or a custom delegate kernel. These methods provide access to the context in which that custom op or custom delegate kernel occurs, such as access to the input and output tensors for that op, as well as methods for allocating memory buffers and intermediate tensors, etc.

See also TfLiteOpaqueContext, which is an more ABI-stable equivalent.

Public attributes

AcquireSubgraphContext)(struct TfLiteContext *context, int subgraph_index, struct TfLiteContext **acquired_context)
TfLiteStatus(*
Retrieves the corresponding TfLiteContext of a subgraph that the given subgraph_index points to and switches to the delegate context for that subgraph.
AddTensors)(struct TfLiteContext *, int tensors_to_add, int *first_new_tensor_index)
TfLiteStatus(*
Add tensors_to_add tensors, preserving pre-existing Tensor entries.
AllocateBufferForEval)(struct TfLiteContext *ctx, size_t bytes, void **ptr)
TfLiteStatus(*
Allocate a buffer which will be deallocated right after invoke phase.
AllocatePersistentBuffer)(struct TfLiteContext *ctx, size_t bytes)
void *(*
Allocate persistent buffer which has the same life time as the interpreter.
GetEvalTensor)(const struct TfLiteContext *context, int tensor_idx)
Returns a TfLiteEvalTensor struct for a given index.
GetExecutionPlan)(struct TfLiteContext *context, TfLiteIntArray **execution_plan)
TfLiteStatus(*
The execution plan contains a list of the node indices in execution order.
GetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType)
Access external contexts by type.
GetModelMetadata)(const struct TfLiteContext *context, const char *name, const char **ptr, size_t *bytes)
TfLiteStatus(*
Retrieves named metadata buffer from the TFLite model.
GetNodeAndRegistration)(struct TfLiteContext *, int node_index, TfLiteNode **node, struct TfLiteRegistration **registration)
TfLiteStatus(*
Get a Tensor node by node_index.
GetScratchBuffer)(struct TfLiteContext *ctx, int buffer_idx)
void *(*
Get the scratch buffer pointer.
GetTensor)(const struct TfLiteContext *context, int tensor_idx)
Returns a TfLiteTensor struct for a given index.
PreviewDelegatePartitioning)(struct TfLiteContext *context, const TfLiteIntArray *nodes_to_replace, TfLiteDelegateParams **partition_params_array, int *num_partitions)
TfLiteStatus(*
This method provides a preview of post-delegation partitioning.
ReleaseSubgraphContext)(struct TfLiteContext *context, int subgraph_index)
TfLiteStatus(*
Releases the subgraph context by switching back to the TFLite kernel context for the subgraph that the given subgraph_index points to.
ReplaceNodeSubsetsWithDelegateKernels)(struct TfLiteContext *, struct TfLiteRegistration registration, const TfLiteIntArray *nodes_to_replace, struct TfLiteDelegate *delegate)
TfLiteStatus(*
Replace ops with one or more stub delegate operations.
ReportError)(struct TfLiteContext *, const char *msg,...)
void(*
Request that an error be reported with format string msg.
RequestScratchBufferInArena)(struct TfLiteContext *ctx, size_t bytes, int *buffer_idx)
TfLiteStatus(*
Request a scratch buffer in the arena through static memory planning.
ResizeTensor)(struct TfLiteContext *, TfLiteTensor *tensor, TfLiteIntArray *new_size)
TfLiteStatus(*
Request memory pointer be resized.
ResizeTensorExplicit)(struct TfLiteContext *ctx, TfLiteTensor *tensor, int dims, const int *shape)
TfLiteStatus(*
Resize the memory pointer of the tensor.
SetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType, TfLiteExternalContext *)
void(*
Set the value of a external context.
allow_fp32_relax_to_fp16
bool
Flag for allowing float16 precision for FP32 calculation.
impl_
void *
opaque full context ptr (an opaque c++ data structure)
profiler
void *
Pointer to the op-level profiler, if set; nullptr otherwise.
recommended_num_threads
int
Number of threads that are recommended to subsystems like gemmlowp and eigen.
tensors
An array of tensors in the interpreter context (of length tensors_size)
tensors_size
size_t
Number of tensors in the context.

Public attributes

AcquireSubgraphContext

TfLiteStatus(* TfLiteContext::AcquireSubgraphContext)(struct TfLiteContext *context, int subgraph_index, struct TfLiteContext **acquired_context)

Retrieves the corresponding TfLiteContext of a subgraph that the given subgraph_index points to and switches to the delegate context for that subgraph.

If an invalid subgraph index is given, returns kTfLiteError.

NOTE: This function is expected to be paired with ReleaseSubgraphContext() once the delegate preparation is done and/or the delegate context functions are no longer needed.

WARNING: This is an experimental interface that is subject to change.

AddTensors

TfLiteStatus(* TfLiteContext::AddTensors)(struct TfLiteContext *, int tensors_to_add, int *first_new_tensor_index)

Add tensors_to_add tensors, preserving pre-existing Tensor entries.

If non-null, the value pointed to by first_new_tensor_index will be set to the index of the first new tensor.

AllocateBufferForEval

TfLiteStatus(* TfLiteContext::AllocateBufferForEval)(struct TfLiteContext *ctx, size_t bytes, void **ptr)

Allocate a buffer which will be deallocated right after invoke phase.

The memory is allocated from heap in TFL, and from volatile arena in TFLM. This method is only available in invoke stage.

NOTE: If possible use RequestScratchBufferInArena method to avoid memory allocation during inference time.

WARNING: This is an experimental interface that is subject to change.

AllocatePersistentBuffer

void *(* TfLiteContext::AllocatePersistentBuffer)(struct TfLiteContext *ctx, size_t bytes)

Allocate persistent buffer which has the same life time as the interpreter.

Returns nullptr on failure. The memory is allocated from heap for TFL, and from tail in TFLM. This method is only available in Init or Prepare stage.

WARNING: This is an experimental interface that is subject to change.

GetEvalTensor

TfLiteEvalTensor *(* TfLiteContext::GetEvalTensor)(const struct TfLiteContext *context, int tensor_idx)

Returns a TfLiteEvalTensor struct for a given index.

WARNING: This is an experimental interface that is subject to change.

WARNING: This method may not be available on all platforms.

GetExecutionPlan

TfLiteStatus(* TfLiteContext::GetExecutionPlan)(struct TfLiteContext *context, TfLiteIntArray **execution_plan)

The execution plan contains a list of the node indices in execution order.

execution_plan->size is the current number of nodes. And, execution_plan->data[0] is the first node that needs to be run. TfLiteDelegates can traverse the current execution plan by iterating through each member of this array and using GetNodeAndRegistration() to access details about a node. i.e.

TfLiteIntArray* execution_plan;
TF_LITE_ENSURE_STATUS(context->GetExecutionPlan(context,
                                                &execution_plan));
for (int exec_index = 0; exec_index < execution_plan->size;
      exec_index++) {
   int node_index = execution_plan->data[exec_index];
   TfLiteNode* node;
   TfLiteRegistration* reg;
   context->GetNodeAndRegistration(context, node_index, &node, ®);
}

Note: the memory pointed by '*execution_plan is OWNED by TfLite runtime. Future calls to GetExecutionPlan invalidates earlier outputs. The following code snippet shows the issue of such an invocation pattern. After calling CheckNode, subsequent access to plan_1st is undefined.

void CheckNode(const TfLiteNode* node) {
  ...
  TfLiteIntArray* plan_2nd;
  TF_LITE_ENSURE_STATUS(
      context->GetExecutionPlan(context, &plan_2nd)
  );
  ...
}

TfLiteIntArray* plan_1st;
TF_LITE_ENSURE_STATUS(context->GetExecutionPlan(context, &plan_1st));
for (int exec_index = 0; exec_index < plan_1st->size; exec_index++) {
   int node_index = plan_1st->data[exec_index];
   TfLiteNode* node;
   TfLiteRegistration* reg;
   context->GetNodeAndRegistration(context, node_index, &node, ®);
   CheckNode(node);
}

WARNING: This is an experimental interface that is subject to change.

GetExternalContext

TfLiteExternalContext *(* TfLiteContext::GetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType)

Access external contexts by type.

WARNING: This is an experimental interface that is subject to change.

GetModelMetadata

TfLiteStatus(* TfLiteContext::GetModelMetadata)(const struct TfLiteContext *context, const char *name, const char **ptr, size_t *bytes)

Retrieves named metadata buffer from the TFLite model.

Returns kTfLiteOk if metadata is successfully obtained from the flatbuffer Model: that is, there exists a metadata entry with given name string. (see TFLite's schema.fbs). The corresponding buffer information is populated in ptr & bytes. The data from ptr is valid for the lifetime of the Interpreter.

WARNING: This is an experimental interface that is subject to change.

GetNodeAndRegistration

TfLiteStatus(* TfLiteContext::GetNodeAndRegistration)(struct TfLiteContext *, int node_index, TfLiteNode **node, struct TfLiteRegistration **registration)

Get a Tensor node by node_index.

WARNING: This is an experimental interface that is subject to change.

GetScratchBuffer

void *(* TfLiteContext::GetScratchBuffer)(struct TfLiteContext *ctx, int buffer_idx)

Get the scratch buffer pointer.

This method is only available in Eval stage.

WARNING: This is an experimental interface that is subject to change.

GetTensor

TfLiteTensor *(* TfLiteContext::GetTensor)(const struct TfLiteContext *context, int tensor_idx)

Returns a TfLiteTensor struct for a given index.

WARNING: This is an experimental interface that is subject to change.

WARNING: This method may not be available on all platforms.

PreviewDelegatePartitioning

TfLiteStatus(* TfLiteContext::PreviewDelegatePartitioning)(struct TfLiteContext *context, const TfLiteIntArray *nodes_to_replace, TfLiteDelegateParams **partition_params_array, int *num_partitions)

This method provides a preview of post-delegation partitioning.

Each TfLiteDelegateParams in the referenced array corresponds to one instance of the delegate kernel. Example usage:

TfLiteIntArray* nodes_to_replace = ...;
TfLiteDelegateParams* params_array;
int num_partitions = 0;
TF_LITE_ENSURE_STATUS(context->PreviewDelegatePartitioning(
   context, delegate, nodes_to_replace, &params_array,
   &num_partitions));
for (int idx = 0; idx < num_partitions; idx++) {
   const auto& partition_params = params_array[idx];
   ...
}

NOTE: The context owns the memory referenced by partition_params_array. It will be cleared with another call to PreviewDelegatePartitioning, or after TfLiteDelegateParams::Prepare returns.

WARNING: This is an experimental interface that is subject to change.

ReleaseSubgraphContext

TfLiteStatus(* TfLiteContext::ReleaseSubgraphContext)(struct TfLiteContext *context, int subgraph_index)

Releases the subgraph context by switching back to the TFLite kernel context for the subgraph that the given subgraph_index points to.

NOTE: This function is expected to be used after AcquireSubgraphContext() once the delegate preparation is done and/or the delegate context functions are no longer needed.

WARNING: This is an experimental interface that is subject to change.

ReplaceNodeSubsetsWithDelegateKernels

TfLiteStatus(* TfLiteContext::ReplaceNodeSubsetsWithDelegateKernels)(struct TfLiteContext *, struct TfLiteRegistration registration, const TfLiteIntArray *nodes_to_replace, struct TfLiteDelegate *delegate)

Replace ops with one or more stub delegate operations.

This function does not take ownership of nodes_to_replace.

ReportError

void(* TfLiteContext::ReportError)(struct TfLiteContext *, const char *msg,...)

Request that an error be reported with format string msg.

RequestScratchBufferInArena

TfLiteStatus(* TfLiteContext::RequestScratchBufferInArena)(struct TfLiteContext *ctx, size_t bytes, int *buffer_idx)

Request a scratch buffer in the arena through static memory planning.

This method is only available in Prepare stage and the buffer is allocated by the interpreter between Prepare and Eval stage. In Eval stage, GetScratchBuffer API can be used to fetch the address.

WARNING: This is an experimental interface that is subject to change.

ResizeTensor

TfLiteStatus(* TfLiteContext::ResizeTensor)(struct TfLiteContext *, TfLiteTensor *tensor, TfLiteIntArray *new_size)

Request memory pointer be resized.

Updates dimensions on the tensor. NOTE: ResizeTensor takes ownership of newSize.

ResizeTensorExplicit

TfLiteStatus(* TfLiteContext::ResizeTensorExplicit)(struct TfLiteContext *ctx, TfLiteTensor *tensor, int dims, const int *shape)

Resize the memory pointer of the tensor.

This method behaves the same as ResizeTensor, except that it makes a copy of the shape array internally so the shape array could be deallocated right afterwards.

WARNING: This is an experimental interface that is subject to change.

SetExternalContext

void(* TfLiteContext::SetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType, TfLiteExternalContext *)

Set the value of a external context.

Does not take ownership of the pointer.

WARNING: This is an experimental interface that is subject to change.

allow_fp32_relax_to_fp16

bool TfLiteContext::allow_fp32_relax_to_fp16

Flag for allowing float16 precision for FP32 calculation.

default: false.

WARNING: This is an experimental API and subject to change.

impl_

void * TfLiteContext::impl_

opaque full context ptr (an opaque c++ data structure)

profiler

void * TfLiteContext::profiler

Pointer to the op-level profiler, if set; nullptr otherwise.

int TfLiteContext::recommended_num_threads

Number of threads that are recommended to subsystems like gemmlowp and eigen.

tensors

TfLiteTensor * TfLiteContext::tensors

An array of tensors in the interpreter context (of length tensors_size)

tensors_size

size_t TfLiteContext::tensors_size

Number of tensors in the context.