Introducing Google AI Edge Portal: Benchmark Edge AI at scale. Sign-up to request access during private preview.

litert::CompiledModel

#include <litert_compiled_model.h>

A high-level inference API for LiteRT.

Summary

The CompiledModel is created by providing a model with compilation options. Internally, it instantiates a runtime and applies delegates mapped to the compilation options. It supports querying buffer requirements to create input/output TensorBuffers and invoking the model with them.

Example User Flow:

Create a CompiledModel.
Query the model's input/output requirements.
Create input/output TensorBuffers.
Fill the input TensorBuffers with input data.
Invoke the model with the input/output TensorBuffers.
Evaluate the output TensorBuffers.

Inheritance

Inherits from: internal::Handle< LiteRtCompiledModel, LiteRtDestroyCompiledModel >

Constructors and Destructors
`CompiledModel()`

Friend classes
`::mediapipe::InferenceRunnerLiteRt`	`friend class`
`benchmark::BenchmarkLiteRtModel`	`friend class`
`compiled_model_wrapper::CompiledModelWrapper`	`friend class`
`lm::AudioLiteRtCompiledModelExecutor`	`friend class`
`lm::EmbeddingLookupText`	`friend class`
`lm::EndOfMultiModalEmbedding`	`friend class`
`lm::FrontendModelWrapper`	`friend class`
`lm::LlmLiteRtCompiledModelExecutorBase`	`friend class`
`lm::LlmLiteRtCompiledModelExecutorDynamic`	`friend class`
`lm::LlmLiteRtCompiledModelExecutorStatic`	`friend class`
`lm::LlmLiteRtNpuCompiledModelExecutor`	`friend class`
`lm::VisionLiteRtCompiledModelExecutor`	`friend class`

Protected attributes
`check_cancelled_func_`	`absl::AnyInvocable< bool()>`
`env_`	`LiteRtEnvironment`
`model_`	`Model`

Public functions
`ClearErrors() const`	`Expected< void >` Clears all errors from the error reporter.
`CreateInputBuffer(absl::string_view signature_name, absl::string_view input_name) const`	`Expected< TensorBuffer >` Creates an input tensor buffer for the given signature and input name.
`CreateInputBuffer(absl::string_view input_name) const`	`Expected< TensorBuffer >` Creates an input tensor buffer for the default signature and a given input name.
`CreateInputBuffers(absl::string_view signature_name) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create input tensor buffers for a given signature.
`CreateInputBuffers(size_t signature_index) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create input tensor buffers for a given signature.
`CreateInputBuffers() const`	`Expected< std::vector< TensorBuffer > >` A helper function to create input tensor buffers for the default signature.
`CreateOutputBuffer(absl::string_view signature_name, absl::string_view output_name) const`	`Expected< TensorBuffer >` Creates an output tensor buffer for the given signature and output name.
`CreateOutputBuffer(absl::string_view output_name) const`	`Expected< TensorBuffer >` Creates an output tensor buffer for the default signature and a given output name.
`CreateOutputBuffers(absl::string_view signature_name) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create output tensor buffers for a given signature.
`CreateOutputBuffers(size_t signature_index) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create output tensor buffers for a given signature.
`CreateOutputBuffers() const`	`Expected< std::vector< TensorBuffer > >` A helper function to create output tensor buffers for the default signature.
`GetErrorMessages() const`	`Expected< std::string >` Gets all error messages from the error reporter as a single string.
`GetInputBufferRequirements(absl::string_view signature_name, absl::string_view input_name)`	`Expected< TensorBufferRequirements >` Gets input buffer requirements for the given signature and input name.
`GetInputBufferRequirements(size_t signature_index, size_t input_index) const`	`Expected< TensorBufferRequirements >` Returns the buffer requirements for the n-th input tensor.
`GetInputBufferRequirements(size_t signature_index, absl::string_view input_name) const`	`Expected< TensorBufferRequirements >` An overload of `GetInputBufferRequirements` that takes an input tensor name.
`GetInputBufferRequirements(size_t input_index) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for the n-th input tensor.
`GetInputBufferRequirements(absl::string_view input_name) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for a given input name.
`GetInputTensorType(size_t signature_index, size_t input_index) const`	`Expected< RankedTensorType >` Returns the tensor type for the n-th input tensor.
`GetInputTensorType(size_t signature_index, absl::string_view input_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given input tensor name.
`GetInputTensorType(absl::string_view signature_key, absl::string_view input_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given input tensor name.
`GetInputTensorType(absl::string_view input_name) const`	`Expected< RankedTensorType >` Gets the input tensor type of the default signature for a given input name.
`GetNumSignatures() const`	`size_t` Returns the number of signatures defined in the model.
`GetOutputBufferRequirements(absl::string_view signature_name, absl::string_view output_name)`	`Expected< TensorBufferRequirements >` Gets output buffer requirements for the given signature and output name.
`GetOutputBufferRequirements(size_t signature_index, size_t output_index) const`	`Expected< TensorBufferRequirements >` Returns the buffer requirements for the given output tensor.
`GetOutputBufferRequirements(size_t signature_index, absl::string_view output_name) const`	`Expected< TensorBufferRequirements >` An overload of `GetOutputBufferRequirements` that takes an output tensor name.
`GetOutputBufferRequirements(size_t output_index) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for the n-th input tensor.
`GetOutputBufferRequirements(absl::string_view output_name) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for a given input name.
`GetOutputTensorLayouts(size_t signature_index, bool update_allocation) const`	`Expected< std::vector< Layout > >` Returns the layouts of all output tensors for a given signature index.
`GetOutputTensorType(size_t signature_index, size_t output_index) const`	`Expected< RankedTensorType >` Returns the tensor type for the n-th output tensor.
`GetOutputTensorType(size_t signature_index, absl::string_view output_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given output tensor name.
`GetOutputTensorType(absl::string_view signature_key, absl::string_view output_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given output tensor name.
`GetOutputTensorType(absl::string_view output_name) const`	`Expected< RankedTensorType >` Gets the output tensor type of the default signature for a given output name.
`GetProfiler()`	`Expected< Profiler >` Returns the profiler used by the compiled model.
`GetSignature(size_t signature_index) const`	`Expected< SimpleSignature >` Returns the signature at the given index.
`GetSignatureIndex(absl::string_view signature_key) const`	`Expected< size_t >` Returns the signature index for a given signature key.
`GetSignatureInputNames(size_t signature_index) const`	`Expected< std::vector< absl::string_view > >` Returns the list of input names defined in the signature.
`GetSignatureInputNames() const`	`Expected< std::vector< absl::string_view > >` Returns the list of input names defined in the signature.
`GetSignatureInputNames(absl::string_view signature_key) const`	`Expected< std::vector< absl::string_view > >` Returns the list of input names defined in the signature.
`GetSignatureKeys() const`	`Expected< std::vector< absl::string_view > >` Returns the list of signature key names defined in the signature.
`GetSignatureOutputNames(size_t signature_index) const`	`Expected< std::vector< absl::string_view > >` Returns the list of output names defined in the signature.
`GetSignatureOutputNames() const`	`Expected< std::vector< absl::string_view > >` Returns the list of output names defined in the signature.
`GetSignatureOutputNames(absl::string_view signature_key) const`	`Expected< std::vector< absl::string_view > >` Returns the list of output names defined in the signature.
`GetSignatures() const`	`Expected< std::vector< SimpleSignature > >` Returns the list of signatures defined in the model.
`IsFullyAccelerated()`	`Expected< bool >` Returns `true` if the compiled model is fully accelerated with the given hardware accelerators.
`ReportError(const char *format, Args... args) const`	`Expected< void >` Reports an error to the compiled model's error reporter.
`ResizeInputTensor(size_t signature_index, size_t input_index, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor to support dynamic shapes.
`ResizeInputTensor(size_t signature_index, absl::string_view input_name, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature.
`ResizeInputTensor(absl::string_view signature_name, absl::string_view input_name, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature name.
`ResizeInputTensor(size_t input_index, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by index.
`ResizeInputTensor(absl::string_view input_name, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by name.
`ResizeInputTensorNonStrict(size_t signature_index, size_t input_index, absl::Span< const int > dims)`	`Expected< void >` Non-strict variants mirror TFLite's `ResizeInputTensor` behavior, allowing arbitrary shape updates when backends support them.
`ResizeInputTensorNonStrict(size_t signature_index, absl::string_view input_name, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature.
`ResizeInputTensorNonStrict(absl::string_view signature_name, absl::string_view input_name, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature name.
`ResizeInputTensorNonStrict(size_t input_index, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by index.
`ResizeInputTensorNonStrict(absl::string_view input_name, absl::Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by name.
`Run(size_t signature_index, absl::Span< const TensorBuffer > input_buffers, absl::Span< const TensorBuffer > output_buffers) const`	`Expected< void >` Runs the model for a given signature index synchronously with the provided input/output `TensorBuffer`s.
`Run(absl::Span< const TensorBuffer > input_buffers, absl::Span< const TensorBuffer > output_buffers) const`	`Expected< void >` Runs the model for the default signature synchronously with the provided input/output `TensorBuffer`s.
`Run(absl::string_view signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers) const`	`Expected< void >` Runs the model for a given signature key synchronously with the provided input/output `TensorBuffer`s.
`Run(absl::string_view signature_key, const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map, const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map) const`	`Expected< void >` Runs the model for a given signature key synchronously with the provided input/output `TensorBuffer` map.
`Run(const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map, const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map) const`	`Expected< void >` Runs the model for the default signature synchronously with the provided input/output `TensorBuffer` map.
`RunAsync(size_t signature_index, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async) const`	`Expected< void >` Runs the model for a given signature index asynchronously, if possible, with the provided input/output `TensorBuffer`s.
`RunAsync(const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async) const`	`Expected< void >` Runs the model for the default signature asynchronously, if possible, with the provided input/output `TensorBuffer`s.
`RunAsync(absl::string_view signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async) const`	`Expected< void >` Runs the model for a given signature key asynchronously, if possible, with the provided input/output `TensorBuffer`s.
`RunAsync(absl::string_view signature_key, const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map, const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map, bool & async) const`	`Expected< void >` Runs the model for a given signature key asynchronously, if possible, with the provided input/output `TensorBuffer` map.
`SetCancellationFunction(void data, bool()(void *) check_cancelled_func)`	`void` Sets a callback function that will be called after every node/op during model execution to check if the execution should be cancelled.
`SetCancellationFunction(absl::AnyInvocable< bool()> check_cancelled_func)`	`void` Sets a callback function for checking cancellation during execution.

Public static functions
`Create(litert::Environment & env, const std::string & model_filename, Options & compilation_options)`	`Expected< CompiledModel >` Creates a `CompiledModel` from a TFLite file.
`Create(litert::Environment & env, BufferRef< uint8_t > model_buffer, Options & compilation_options)`	`Expected< CompiledModel >` An overload of `Create` that takes a buffer reference to the model instead of a filename.
`Create(litert::Environment & env, const std::string & model_filename, litert::HwAccelerators hardware_accelerators)`	`Expected< CompiledModel >` A simplified version of `Create` that uses default compilation options.
`Create(litert::Environment & env, BufferRef< uint8_t > model_buffer, litert::HwAccelerators hardware_accelerators)`	`Expected< CompiledModel >` An overload of `Create` that takes a buffer reference to the model instead of a filename.
`DefaultSignatureKey()`	`absl::string_view` Returns the default signature key of the model.
`WrapCObject(LiteRtModel litert_model, LiteRtCompiledModel compiled_model, OwnHandle owned)`	`CompiledModel`

Protected static functions
`CheckCancelledWrapper(void *data)`	`bool`
`Create(litert::Environment & env, const litert::Model & model, Options & compilation_options)`	`Expected< CompiledModel >`
`Create(litert::Environment & env, const litert::Model & model, litert::HwAccelerators hardware_accelerators)`	`Expected< CompiledModel >`
`CreateBufferImpl(const Environment & env, const TensorBufferRequirements & buffer_requirements, const RankedTensorType & tensor_type)`	`Expected< TensorBuffer >` Creates a `TensorBuffer` with the given buffer requirements and tensor type.

Protected functions
`CompiledModel(LiteRtModel litert_model, OwnHandle model_owned, LiteRtCompiledModel compiled_model, OwnHandle owned)`	Constructs a `CompiledModel` instance.
`CreateInputOutputBuffer(size_t signature_index, absl::string_view tensor_name, bool is_input) const`	`Expected< TensorBuffer >` Creates a `TensorBuffer` for a given signature index and tensor name.
`CreateInputOutputBuffer(absl::string_view signature_name, absl::string_view tensor_name, bool is_input) const`	`Expected< TensorBuffer >` Creates a `TensorBuffer` for a given signature and tensor name.
`CreateInputOutputBuffers(size_t signature_index, bool is_input) const`	`Expected< std::vector< TensorBuffer > >` Creates a vector of `TensorBuffer`s for a given signature subgraph.
`FindInputIndex(size_t signature_index, absl::string_view input_name) const`	`Expected< size_t >` Returns the signature input index for a given input tensor name.
`FindOutputIndex(size_t signature_index, absl::string_view output_name) const`	`Expected< size_t >` Returns the signature output index for a given output tensor name.
`GetEnvironment() const`	`Expected< Environment >` Returns the environment used to create this compiled model.
`RunCApiHelper(LiteRtParamIndex signature_index, size_t num_input_buffers, LiteRtTensorBuffer input_buffers, size_t num_output_buffers, LiteRtTensorBuffer output_buffers, bool & async) const`	`Expected< void >`
`RunHelper(size_t signature_index, absl::Span< const TensorBuffer > input_buffers, absl::Span< const TensorBuffer > output_buffers, bool & async) const`	`Expected< void >`
`RunMapHelper(absl::string_view signature_key, const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map, const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map, bool & async) const`	`Expected< void >`
`RunMapWithIndexHelper(size_t signature_index, const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map, const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map, bool & async) const`	`Expected< void >`

Friend classes

::mediapipe::InferenceRunnerLiteRt

friend class ::mediapipe::InferenceRunnerLiteRt

benchmark::BenchmarkLiteRtModel

friend class benchmark::BenchmarkLiteRtModel

compiled_model_wrapper::CompiledModelWrapper

friend class compiled_model_wrapper::CompiledModelWrapper

lm::AudioLiteRtCompiledModelExecutor

friend class lm::AudioLiteRtCompiledModelExecutor

lm::EmbeddingLookupText

friend class lm::EmbeddingLookupText

lm::EndOfMultiModalEmbedding

friend class lm::EndOfMultiModalEmbedding

lm::FrontendModelWrapper

friend class lm::FrontendModelWrapper

lm::LlmLiteRtCompiledModelExecutorBase

friend class lm::LlmLiteRtCompiledModelExecutorBase

lm::LlmLiteRtCompiledModelExecutorDynamic

friend class lm::LlmLiteRtCompiledModelExecutorDynamic

lm::LlmLiteRtCompiledModelExecutorStatic

friend class lm::LlmLiteRtCompiledModelExecutorStatic

lm::LlmLiteRtNpuCompiledModelExecutor

friend class lm::LlmLiteRtNpuCompiledModelExecutor

lm::VisionLiteRtCompiledModelExecutor

friend class lm::VisionLiteRtCompiledModelExecutor

Protected attributes

check_cancelled_func_

absl::AnyInvocable< bool()> check_cancelled_func_

env_

LiteRtEnvironment env_

model_

Model model_

Public functions

ClearErrors

Expected< void > ClearErrors() const

Clears all errors from the error reporter.

Note: This only works if the compiled model uses BufferErrorReporter, not StderrReporter.

CompiledModel

 CompiledModel()=default

CreateInputBuffer

Expected< TensorBuffer > CreateInputBuffer(
  absl::string_view signature_name,
  absl::string_view input_name
) const

Creates an input tensor buffer for the given signature and input name.

CreateInputBuffer

Expected< TensorBuffer > CreateInputBuffer(
  absl::string_view input_name
) const

Creates an input tensor buffer for the default signature and a given input name.

CreateInputBuffers

Expected< std::vector< TensorBuffer > > CreateInputBuffers(
  absl::string_view signature_name
) const

A helper function to create input tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the input tensor buffers.

CreateInputBuffers

Expected< std::vector< TensorBuffer > > CreateInputBuffers(
  size_t signature_index
) const

A helper function to create input tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the input tensor buffers.

CreateInputBuffers

Expected< std::vector< TensorBuffer > > CreateInputBuffers() const

A helper function to create input tensor buffers for the default signature.

It uses BufferRequirements and RankedTensorType to create the input tensor buffers.

CreateOutputBuffer

Expected< TensorBuffer > CreateOutputBuffer(
  absl::string_view signature_name,
  absl::string_view output_name
) const

Creates an output tensor buffer for the given signature and output name.

CreateOutputBuffer

Expected< TensorBuffer > CreateOutputBuffer(
  absl::string_view output_name
) const

Creates an output tensor buffer for the default signature and a given output name.

CreateOutputBuffers

Expected< std::vector< TensorBuffer > > CreateOutputBuffers(
  absl::string_view signature_name
) const

A helper function to create output tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the output tensor buffers.

CreateOutputBuffers

Expected< std::vector< TensorBuffer > > CreateOutputBuffers(
  size_t signature_index
) const

A helper function to create output tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the output tensor buffers.

CreateOutputBuffers

Expected< std::vector< TensorBuffer > > CreateOutputBuffers() const

A helper function to create output tensor buffers for the default signature.

It uses BufferRequirements and RankedTensorType to create the output tensor buffers.

GetErrorMessages

Expected< std::string > GetErrorMessages() const

Gets all error messages from the error reporter as a single string.

Note: This only works if the compiled model uses BufferErrorReporter, not StderrReporter. The C++ wrapper automatically manages memory using RAII.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  absl::string_view signature_name,
  absl::string_view input_name
)

Gets input buffer requirements for the given signature and input name.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  size_t signature_index,
  size_t input_index
) const

Returns the buffer requirements for the n-th input tensor.

The returned TensorBufferRequirements is used to create the input tensor buffer.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  size_t signature_index,
  absl::string_view input_name
) const

An overload of GetInputBufferRequirements that takes an input tensor name.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  size_t input_index
) const

Gets input buffer requirements of the default signature for the n-th input tensor.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  absl::string_view input_name
) const

Gets input buffer requirements of the default signature for a given input name.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  size_t signature_index,
  size_t input_index
) const

Returns the tensor type for the n-th input tensor.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  size_t signature_index,
  absl::string_view input_name
) const

Returns the tensor type for a given input tensor name.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  absl::string_view signature_key,
  absl::string_view input_name
) const

Returns the tensor type for a given input tensor name.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  absl::string_view input_name
) const

Gets the input tensor type of the default signature for a given input name.

GetNumSignatures

size_t GetNumSignatures() const

Returns the number of signatures defined in the model.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  absl::string_view signature_name,
  absl::string_view output_name
)

Gets output buffer requirements for the given signature and output name.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  size_t signature_index,
  size_t output_index
) const

Returns the buffer requirements for the given output tensor.

The returned TensorBufferRequirements is used to create the output tensor buffer.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  size_t signature_index,
  absl::string_view output_name
) const

An overload of GetOutputBufferRequirements that takes an output tensor name.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  size_t output_index
) const

Gets input buffer requirements of the default signature for the n-th input tensor.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  absl::string_view output_name
) const

Gets input buffer requirements of the default signature for a given input name.

GetOutputTensorLayouts

Expected< std::vector< Layout > > GetOutputTensorLayouts(
  size_t signature_index,
  bool update_allocation
) const

Returns the layouts of all output tensors for a given signature index.

If update_allocation is true, the tensor allocations will be updated to the current state of the compiled model.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  size_t signature_index,
  size_t output_index
) const

Returns the tensor type for the n-th output tensor.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  size_t signature_index,
  absl::string_view output_name
) const

Returns the tensor type for a given output tensor name.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  absl::string_view signature_key,
  absl::string_view output_name
) const

Returns the tensor type for a given output tensor name.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  absl::string_view output_name
) const

Gets the output tensor type of the default signature for a given output name.

GetProfiler

Expected< Profiler > GetProfiler()

Returns the profiler used by the compiled model.

The returned Profiler does not own the underlying LiteRtProfiler.

GetSignature

Expected< SimpleSignature > GetSignature(
  size_t signature_index
) const

Returns the signature at the given index.

GetSignatureIndex

Expected< size_t > GetSignatureIndex(
  absl::string_view signature_key
) const

Returns the signature index for a given signature key.

Returns 0 if the signature key is empty.

GetSignatureInputNames

Expected< std::vector< absl::string_view > > GetSignatureInputNames(
  size_t signature_index
) const

Returns the list of input names defined in the signature.

GetSignatureInputNames

Expected< std::vector< absl::string_view > > GetSignatureInputNames() const

Returns the list of input names defined in the signature.

GetSignatureInputNames

Expected< std::vector< absl::string_view > > GetSignatureInputNames(
  absl::string_view signature_key
) const

Returns the list of input names defined in the signature.

GetSignatureKeys

Expected< std::vector< absl::string_view > > GetSignatureKeys() const

Returns the list of signature key names defined in the signature.

GetSignatureOutputNames

Expected< std::vector< absl::string_view > > GetSignatureOutputNames(
  size_t signature_index
) const

Returns the list of output names defined in the signature.

GetSignatureOutputNames

Expected< std::vector< absl::string_view > > GetSignatureOutputNames() const

Returns the list of output names defined in the signature.

GetSignatureOutputNames

Expected< std::vector< absl::string_view > > GetSignatureOutputNames(
  absl::string_view signature_key
) const

Returns the list of output names defined in the signature.

GetSignatures

Expected< std::vector< SimpleSignature > > GetSignatures() const

Returns the list of signatures defined in the model.

IsFullyAccelerated

Expected< bool > IsFullyAccelerated()

Returns true if the compiled model is fully accelerated with the given hardware accelerators.

ReportError

Expected< void > ReportError(
  const char *format,
  Args... args
) const

Reports an error to the compiled model's error reporter.

Supports printf-style formatting for error messages.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  size_t signature_index,
  size_t input_index,
  absl::Span< const int > dims
)

Resizes the specified input tensor to support dynamic shapes.

This function mirrors TFLite's ResizeInputTensorStrict API and requires the tensor signature to include dynamic dimensions. After calling this function, the compiled model will reallocate internal buffers as needed to accommodate the new tensor shape. For models that need relaxed validation, use ResizeInputTensorNonStrict.

Details

Parameters

`signature_index`	The index of the signature in the model.
`input_index`	The index of the input tensor in the signature.
`dims`	The new dimensions for the input tensor.

Returns

Success if the resize operation completes successfully, or an error with an appropriate status code on failure.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  size_t signature_index,
  absl::string_view input_name,
  absl::Span< const int > dims
)

Resizes the specified input tensor by name for the given signature.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  absl::string_view signature_name,
  absl::string_view input_name,
  absl::Span< const int > dims
)

Resizes the specified input tensor by name for the given signature name.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  size_t input_index,
  absl::Span< const int > dims
)

Resizes the specified input tensor of the default signature by index.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  absl::string_view input_name,
  absl::Span< const int > dims
)

Resizes the specified input tensor of the default signature by name.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  size_t signature_index,
  size_t input_index,
  absl::Span< const int > dims
)

Non-strict variants mirror TFLite's ResizeInputTensor behavior, allowing arbitrary shape updates when backends support them.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  size_t signature_index,
  absl::string_view input_name,
  absl::Span< const int > dims
)

Resizes the specified input tensor by name for the given signature.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  absl::string_view signature_name,
  absl::string_view input_name,
  absl::Span< const int > dims
)

Resizes the specified input tensor by name for the given signature name.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  size_t input_index,
  absl::Span< const int > dims
)

Resizes the specified input tensor of the default signature by index.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  absl::string_view input_name,
  absl::Span< const int > dims
)

Resizes the specified input tensor of the default signature by name.

Run

Expected< void > Run(
  size_t signature_index,
  absl::Span< const TensorBuffer > input_buffers,
  absl::Span< const TensorBuffer > output_buffers
) const

Runs the model for a given signature index synchronously with the provided input/output TensorBuffers.

Run

Expected< void > Run(
  absl::Span< const TensorBuffer > input_buffers,
  absl::Span< const TensorBuffer > output_buffers
) const

Runs the model for the default signature synchronously with the provided input/output TensorBuffers.

Run

Expected< void > Run(
  absl::string_view signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers
) const

Runs the model for a given signature key synchronously with the provided input/output TensorBuffers.

Run

Expected< void > Run(
  absl::string_view signature_key,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map
) const

Runs the model for a given signature key synchronously with the provided input/output TensorBuffer map.

If you have bound the input with external buffers through Options, you can skip providing those input buffers in the map.

Run

Expected< void > Run(
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map
) const

Runs the model for the default signature synchronously with the provided input/output TensorBuffer map.

If you have bound the input with external buffers through Options, you can skip providing those input buffers in the map.

RunAsync

Expected< void > RunAsync(
  size_t signature_index,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async
) const

Runs the model for a given signature index asynchronously, if possible, with the provided input/output TensorBuffers.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

RunAsync

Expected< void > RunAsync(
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async
) const

Runs the model for the default signature asynchronously, if possible, with the provided input/output TensorBuffers.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

RunAsync

Expected< void > RunAsync(
  absl::string_view signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async
) const

Runs the model for a given signature key asynchronously, if possible, with the provided input/output TensorBuffers.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

RunAsync

Expected< void > RunAsync(
  absl::string_view signature_key,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map,
  bool & async
) const

Runs the model for a given signature key asynchronously, if possible, with the provided input/output TensorBuffer map.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

SetCancellationFunction

void SetCancellationFunction(
  void *data,
  bool(*)(void *) check_cancelled_func
)

Sets a callback function that will be called after every node/op during model execution to check if the execution should be cancelled.

This behavior is defined here: tflite/core/subgraph.cc;l=1746-1750?q=tflite%20subgraph The callback should return true if execution should be cancelled.

Note: Use either this callback-based mechanism or the non-callback version (see below) with EnableCancellation/Cancel, but not both.

SetCancellationFunction

void SetCancellationFunction(
  absl::AnyInvocable< bool()> check_cancelled_func
)

Sets a callback function for checking cancellation during execution.

The callback will be called periodically during model execution. This is a C++-friendly version of SetCancellationFunction.

Public static functions

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const std::string & model_filename,
  Options & compilation_options
)

Creates a CompiledModel from a TFLite file.

The model is loaded into memory, and the caller takes ownership of the returned CompiledModel object. The caller should keep the model alive until the CompiledModel is destroyed. The provided compilation_options are used for model compilation, and hardware_accelerators is used to select the accelerator, regardless of whether the model is AOT or JIT compiled.

Note: Even if the model is fully AOT-compiled for an NPU, you must specify the NPU accelerator in hardware_accelerators to use it properly.

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  BufferRef< uint8_t > model_buffer,
  Options & compilation_options
)

An overload of Create that takes a buffer reference to the model instead of a filename.

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const std::string & model_filename,
  litert::HwAccelerators hardware_accelerators
)

A simplified version of Create that uses default compilation options.

The provided hardware accelerator is used to select the target accelerator.

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  BufferRef< uint8_t > model_buffer,
  litert::HwAccelerators hardware_accelerators
)

An overload of Create that takes a buffer reference to the model instead of a filename.

DefaultSignatureKey

absl::string_view DefaultSignatureKey()

Returns the default signature key of the model.

WrapCObject

CompiledModel WrapCObject(
  LiteRtModel litert_model,
  LiteRtCompiledModel compiled_model,
  OwnHandle owned
)

Protected static functions

CheckCancelledWrapper

bool CheckCancelledWrapper(
  void *data
)

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const litert::Model & model,
  Options & compilation_options
)

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const litert::Model & model,
  litert::HwAccelerators hardware_accelerators
)

CreateBufferImpl

Expected< TensorBuffer > CreateBufferImpl(
  const Environment & env,
  const TensorBufferRequirements & buffer_requirements,
  const RankedTensorType & tensor_type
)

Creates a TensorBuffer with the given buffer requirements and tensor type.

Protected functions

CompiledModel

 CompiledModel(
  LiteRtModel litert_model,
  OwnHandle model_owned,
  LiteRtCompiledModel compiled_model,
  OwnHandle owned
)

Constructs a CompiledModel instance.

Details

Parameters

`model_owned`	Indicates whether the provided `litert_model` handle is owned by the `CompiledModel`.
`owned`	If `true`, the created object takes ownership of the `compiled_model` handle.

CreateInputOutputBuffer

Expected< TensorBuffer > CreateInputOutputBuffer(
  size_t signature_index,
  absl::string_view tensor_name,
  bool is_input
) const

Creates a TensorBuffer for a given signature index and tensor name.

CreateInputOutputBuffer

Expected< TensorBuffer > CreateInputOutputBuffer(
  absl::string_view signature_name,
  absl::string_view tensor_name,
  bool is_input
) const

Creates a TensorBuffer for a given signature and tensor name.

CreateInputOutputBuffers

Expected< std::vector< TensorBuffer > > CreateInputOutputBuffers(
  size_t signature_index,
  bool is_input
) const

Creates a vector of TensorBuffers for a given signature subgraph.

FindInputIndex

Expected< size_t > FindInputIndex(
  size_t signature_index,
  absl::string_view input_name
) const

Returns the signature input index for a given input tensor name.

FindOutputIndex

Expected< size_t > FindOutputIndex(
  size_t signature_index,
  absl::string_view output_name
) const

Returns the signature output index for a given output tensor name.

GetEnvironment

Expected< Environment > GetEnvironment() const

Returns the environment used to create this compiled model.

The returned Environment does not own the underlying LiteRtEnvironment.

RunCApiHelper

Expected< void > RunCApiHelper(
  LiteRtParamIndex signature_index,
  size_t num_input_buffers,
  LiteRtTensorBuffer *input_buffers,
  size_t num_output_buffers,
  LiteRtTensorBuffer *output_buffers,
  bool & async
) const

RunHelper

Expected< void > RunHelper(
  size_t signature_index,
  absl::Span< const TensorBuffer > input_buffers,
  absl::Span< const TensorBuffer > output_buffers,
  bool & async
) const

RunMapHelper

Expected< void > RunMapHelper(
  absl::string_view signature_key,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map,
  bool & async
) const

RunMapWithIndexHelper

Expected< void > RunMapWithIndexHelper(
  size_t signature_index,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & input_map,
  const absl::flat_hash_map< absl::string_view, TensorBuffer > & output_map,
  bool & async
) const

litert::CompiledModel

Summary

Inheritance

Constructors and Destructors

Friend classes

Protected attributes

Public functions

Public static functions

Protected static functions

Protected functions

Friend classes

::mediapipe::InferenceRunnerLiteRt

benchmark::BenchmarkLiteRtModel

compiled_model_wrapper::CompiledModelWrapper

lm::AudioLiteRtCompiledModelExecutor

lm::EmbeddingLookupText

lm::EndOfMultiModalEmbedding

lm::FrontendModelWrapper

lm::LlmLiteRtCompiledModelExecutorBase

lm::LlmLiteRtCompiledModelExecutorDynamic

lm::LlmLiteRtCompiledModelExecutorStatic

lm::LlmLiteRtNpuCompiledModelExecutor

lm::VisionLiteRtCompiledModelExecutor

Protected attributes

check_cancelled_func_

env_

model_

Public functions

ClearErrors

CompiledModel

CreateInputBuffer

CreateInputBuffer

CreateInputBuffers

CreateInputBuffers

CreateInputBuffers

CreateOutputBuffer

CreateOutputBuffer

CreateOutputBuffers

CreateOutputBuffers

CreateOutputBuffers

GetErrorMessages

GetInputBufferRequirements

GetInputBufferRequirements

GetInputBufferRequirements

GetInputBufferRequirements

GetInputBufferRequirements

GetInputTensorType

GetInputTensorType

GetInputTensorType

GetInputTensorType

GetNumSignatures

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputTensorLayouts

GetOutputTensorType

GetOutputTensorType

GetOutputTensorType

GetOutputTensorType

GetProfiler

GetSignature

GetSignatureIndex

GetSignatureInputNames

GetSignatureInputNames

GetSignatureInputNames

GetSignatureKeys

GetSignatureOutputNames

GetSignatureOutputNames

GetSignatureOutputNames

GetSignatures

IsFullyAccelerated

ReportError

ResizeInputTensor

ResizeInputTensor

ResizeInputTensor

ResizeInputTensor

ResizeInputTensor

ResizeInputTensorNonStrict