Introducing Google AI Edge Portal: Benchmark Edge AI at scale. Sign-up to request access during private preview.

litert::CompiledModel

#include <litert_compiled_model.h>

A high-level inference API for LiteRT.

Summary

The CompiledModel is created by providing a model with compilation options. Internally, it instantiates a runtime and applies delegates mapped to the compilation options. It supports querying buffer requirements to create input/output TensorBuffers and invoking the model with them.

Example User Flow:

Create a CompiledModel.
Query the model's input/output requirements.
Create input/output TensorBuffers.
Fill the input TensorBuffers with input data.
Invoke the model with the input/output TensorBuffers.
Evaluate the output TensorBuffers.

Inheritance

Inherits from: internal::BaseHandle< LiteRtCompiledModel >

Constructors and Destructors
`CompiledModel()`

Friend classes
`::mediapipe::InferenceRunnerLiteRt`	`friend class`
`benchmark::BenchmarkLiteRtModel`	`friend class`
`compiled_model_wrapper::CompiledModelWrapper`	`friend class`
`lm::AudioLiteRtCompiledModelExecutor`	`friend class`
`lm::EmbeddingLookupText`	`friend class`
`lm::EndOfMultiModalEmbedding`	`friend class`
`lm::FrontendModelWrapper`	`friend class`
`lm::LlmLiteRtCompiledModelExecutorBase`	`friend class`
`lm::LlmLiteRtCompiledModelExecutorDynamic`	`friend class`
`lm::LlmLiteRtCompiledModelExecutorStatic`	`friend class`
`lm::LlmLiteRtMtpDrafter`	`friend class`
`lm::LlmLiteRtNpuCompiledModelExecutor`	`friend class`
`lm::VisionLiteRtCompiledModelExecutor`	`friend class`

Protected attributes
`check_cancelled_func_`	`AnyInvocable< bool()>`
`env_`	`internal::EnvironmentHolder`
`model_`	`internal::BaseHandle< LiteRtModel >`
`options_`	`internal::LiteRtOptionsPtr`

Public functions
`ClearErrors() const`	`Expected< void >` Clears all errors from the error reporter.
`ClearSchedulingInfo() const`	`Expected< void >` Clears model-level default scheduling info.
`CreateInputBuffer(StringView signature_name, StringView input_name) const`	`Expected< TensorBuffer >` Creates an input tensor buffer for the given signature and input name.
`CreateInputBuffer(StringView input_name) const`	`Expected< TensorBuffer >` Creates an input tensor buffer for the default signature and a given input name.
`CreateInputBuffers(StringView signature_name) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create input tensor buffers for a given signature.
`CreateInputBuffers(size_t signature_index) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create input tensor buffers for a given signature.
`CreateInputBuffers() const`	`Expected< std::vector< TensorBuffer > >` A helper function to create input tensor buffers for the default signature.
`CreateOutputBuffer(StringView signature_name, StringView output_name) const`	`Expected< TensorBuffer >` Creates an output tensor buffer for the given signature and output name.
`CreateOutputBuffer(StringView output_name) const`	`Expected< TensorBuffer >` Creates an output tensor buffer for the default signature and a given output name.
`CreateOutputBuffers(StringView signature_name) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create output tensor buffers for a given signature.
`CreateOutputBuffers(size_t signature_index) const`	`Expected< std::vector< TensorBuffer > >` A helper function to create output tensor buffers for a given signature.
`CreateOutputBuffers() const`	`Expected< std::vector< TensorBuffer > >` A helper function to create output tensor buffers for the default signature.
`FindSignature(StringView signature_key) const`	`Expected< SimpleSignature >`
`GetDispatchAnnotation(size_t signature_index, StringView key)`	`Expected< std::optional< std::string > >` Gets a dispatch annotation from the compiled model.
`GetDispatchAnnotation(StringView key)`	`Expected< std::optional< std::string > >` Overloaded version for the default signature (index 0).
`GetDispatchAnnotation(StringView signature_name, StringView key)`	`Expected< std::optional< std::string > >` Overloaded version that takes a signature name instead of an index.
`GetErrorMessages() const`	`Expected< std::string >` Gets all error messages from the error reporter as a single string.
`GetInputBufferRequirements(StringView signature_name, StringView input_name)`	`Expected< TensorBufferRequirements >` Gets input buffer requirements for the given signature and input name.
`GetInputBufferRequirements(size_t signature_index, size_t input_index) const`	`Expected< TensorBufferRequirements >` Returns the buffer requirements for the n-th input tensor.
`GetInputBufferRequirements(size_t signature_index, StringView input_name) const`	`Expected< TensorBufferRequirements >` An overload of `GetInputBufferRequirements` that takes an input tensor name.
`GetInputBufferRequirements(size_t input_index) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for the n-th input tensor.
`GetInputBufferRequirements(StringView input_name) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for a given input name.
`GetInputTensorLayout(size_t signature_index, size_t input_index) const`	`Expected< Layout >` Returns the layout of the given input tensor.
`GetInputTensorType(size_t signature_index, size_t input_index) const`	`Expected< RankedTensorType >` Returns the tensor type for the n-th input tensor.
`GetInputTensorType(size_t signature_index, StringView input_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given input tensor name.
`GetInputTensorType(StringView signature_key, StringView input_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given input tensor name.
`GetInputTensorType(StringView input_name) const`	`Expected< RankedTensorType >` Gets the input tensor type of the default signature for a given input name.
`GetNumSignatures() const`	`size_t` Returns the number of signatures defined in the model.
`GetOutputBufferRequirements(StringView signature_name, StringView output_name)`	`Expected< TensorBufferRequirements >` Gets output buffer requirements for the given signature and output name.
`GetOutputBufferRequirements(size_t signature_index, size_t output_index) const`	`Expected< TensorBufferRequirements >` Returns the buffer requirements for the given output tensor.
`GetOutputBufferRequirements(size_t signature_index, StringView output_name) const`	`Expected< TensorBufferRequirements >` An overload of `GetOutputBufferRequirements` that takes an output tensor name.
`GetOutputBufferRequirements(size_t output_index) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for the n-th input tensor.
`GetOutputBufferRequirements(StringView output_name) const`	`Expected< TensorBufferRequirements >` Gets input buffer requirements of the default signature for a given input name.
`GetOutputTensorLayouts(size_t signature_index, bool update_allocation) const`	`Expected< std::vector< Layout > >` Returns the layouts of all output tensors for a given signature index.
`GetOutputTensorType(size_t signature_index, size_t output_index) const`	`Expected< RankedTensorType >` Returns the tensor type for the n-th output tensor.
`GetOutputTensorType(size_t signature_index, StringView output_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given output tensor name.
`GetOutputTensorType(StringView signature_key, StringView output_name) const`	`Expected< RankedTensorType >` Returns the tensor type for a given output tensor name.
`GetOutputTensorType(StringView output_name) const`	`Expected< RankedTensorType >` Gets the output tensor type of the default signature for a given output name.
`GetSignature(size_t signature_index) const`	`Expected< SimpleSignature >` Returns the signature at the given index.
`GetSignatureIndex(StringView signature_key) const`	`Expected< size_t >` Returns the signature index for a given signature key.
`GetSignatureInputNames(size_t signature_index) const`	`Expected< std::vector< StringView > >` Returns the list of input names defined in the signature.
`GetSignatureInputNames() const`	`Expected< std::vector< StringView > >` Returns the list of input names defined in the signature.
`GetSignatureInputNames(StringView signature_key) const`	`Expected< std::vector< StringView > >` Returns the list of input names defined in the signature.
`GetSignatureKeys() const`	`Expected< std::vector< StringView > >` Returns the list of signature key names defined in the signature.
`GetSignatureOutputNames(size_t signature_index) const`	`Expected< std::vector< StringView > >` Returns the list of output names defined in the signature.
`GetSignatureOutputNames() const`	`Expected< std::vector< StringView > >` Returns the list of output names defined in the signature.
`GetSignatureOutputNames(StringView signature_key) const`	`Expected< std::vector< StringView > >` Returns the list of output names defined in the signature.
`GetSignatures() const`	`Expected< std::vector< SimpleSignature > >` Returns the list of signatures defined in the model.
`IsFullyAccelerated()`	`Expected< bool >` Returns `true` if the compiled model is fully accelerated with the given hardware accelerators.
`RemoveDispatchAnnotation(size_t signature_index, StringView key)`	`Expected< void >` Removes a dispatch annotation from the compiled model.
`RemoveDispatchAnnotation(StringView key)`	`Expected< void >` Overloaded version for the default signature (index 0).
`RemoveDispatchAnnotation(StringView signature_name, StringView key)`	`Expected< void >` Overloaded version that takes a signature name instead of an index.
`ReportError(const char *format, Args &&... args) const`	`Expected< void >` Reports an error to the compiled model's error reporter.
`ResizeInputTensor(size_t signature_index, size_t input_index, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor to support dynamic shapes.
`ResizeInputTensor(size_t signature_index, StringView input_name, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature.
`ResizeInputTensor(StringView signature_name, StringView input_name, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature name.
`ResizeInputTensor(size_t input_index, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by index.
`ResizeInputTensor(StringView input_name, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by name.
`ResizeInputTensorNonStrict(size_t signature_index, size_t input_index, Span< const int > dims)`	`Expected< void >` Non-strict variants mirror TFLite's `ResizeInputTensor` behavior, allowing arbitrary shape updates when backends support them.
`ResizeInputTensorNonStrict(size_t signature_index, StringView input_name, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature.
`ResizeInputTensorNonStrict(StringView signature_name, StringView input_name, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor by name for the given signature name.
`ResizeInputTensorNonStrict(size_t input_index, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by index.
`ResizeInputTensorNonStrict(StringView input_name, Span< const int > dims)`	`Expected< void >` Resizes the specified input tensor of the default signature by name.
`Run(size_t signature_index, Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers) const`	`Expected< void >` Runs the model for a given signature index synchronously with the provided input/output `TensorBuffer`s.
`Run(Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers) const`	`Expected< void >` Runs the model for the default signature synchronously with the provided input/output `TensorBuffer`s.
`Run(size_t signature_index, Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers, Options *run_options) const`	`Expected< void >` Runs with per-run options for a given signature index.
`Run(size_t signature_index, Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs with per-request scheduling info for a given signature index.
`Run(Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers, Options *run_options) const`	`Expected< void >` Runs default signature with per-run options.
`Run(Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs default signature with per-request scheduling info.
`Run(StringView signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers) const`	`Expected< void >` Runs the model for a given signature key synchronously with the provided input/output `TensorBuffer`s.
`Run(StringView signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, Options *run_options) const`	`Expected< void >` Runs by signature key with per-run options.
`Run(StringView signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs by signature key with per-request scheduling info.
`Run(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map) const`	`Expected< void >` Runs the model for a given signature key synchronously with the provided input/output `TensorBuffer` map.
`Run(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, Options *run_options) const`	`Expected< void >` Runs by signature key with per-run options using named maps.
`Run(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs by signature key with per-request scheduling info using named maps.
`Run(const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map) const`	`Expected< void >` Runs the model for the default signature synchronously with the provided input/output `TensorBuffer` map.
`Run(const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, Options *run_options) const`	`Expected< void >` Runs default signature with per-run options using named maps.
`Run(const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs default signature with per-request scheduling info using named maps.
`RunAsync(size_t signature_index, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async) const`	`Expected< void >` Runs the model for a given signature index asynchronously, if possible, with the provided input/output `TensorBuffer`s.
`RunAsync(const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async) const`	`Expected< void >` Runs the model for the default signature asynchronously, if possible, with the provided input/output `TensorBuffer`s.
`RunAsync(size_t signature_index, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async, Options *run_options) const`	`Expected< void >` Runs asynchronously with per-run options for a given signature.
`RunAsync(size_t signature_index, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs asynchronously with per-request scheduling info for a given signature.
`RunAsync(const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async, Options *run_options) const`	`Expected< void >` Runs default signature asynchronously with per-run options.
`RunAsync(const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs default signature asynchronously with per-request scheduling info.
`RunAsync(StringView signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async) const`	`Expected< void >` Runs the model for a given signature key asynchronously, if possible, with the provided input/output `TensorBuffer`s.
`RunAsync(StringView signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async, Options *run_options) const`	`Expected< void >` Runs by signature key asynchronously with per-run options.
`RunAsync(StringView signature_key, const std::vector< TensorBuffer > & input_buffers, const std::vector< TensorBuffer > & output_buffers, bool & async, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs by signature key asynchronously with per-request scheduling info.
`RunAsync(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async) const`	`Expected< void >` Runs the model for a given signature key asynchronously, if possible, with the provided input/output `TensorBuffer` map.
`RunAsync(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async, Options *run_options) const`	`Expected< void >` Runs by signature key asynchronously with per-run options using named maps.
`RunAsync(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async, const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Runs by signature key asynchronously with per-request scheduling info using named maps.
`SetCancellationFunction(void data, bool()(void *) check_cancelled_func)`	`void` Sets a callback function that will be called after every node/op during model execution to check if the execution should be cancelled.
`SetCancellationFunction(AnyInvocable< bool()> check_cancelled_func)`	`void` Sets a callback function for checking cancellation during execution.
`SetDispatchAnnotation(size_t signature_index, StringView key, StringView value)`	`Expected< void >` Sets a dispatch annotation on the compiled model.
`SetDispatchAnnotation(StringView key, StringView value)`	`Expected< void >` Overloaded version for the default signature (index 0).
`SetDispatchAnnotation(StringView signature_name, StringView key, StringView value)`	`Expected< void >` Overloaded version that takes a signature name instead of an index.
`SetSchedulingInfo(const LiteRtSchedulingInfo & scheduling_info) const`	`Expected< void >` Sets model-level default scheduling info.

Public static functions
`Create(litert::Environment & env, const std::string & model_filename, Options & compilation_options)`	`Expected< CompiledModel >` Creates a `CompiledModel` from a TFLite file.
`Create(litert::Environment & env, BufferRef< uint8_t > model_buffer, Options & compilation_options)`	`Expected< CompiledModel >` An overload of `Create` that takes a buffer reference to the model instead of a filename.
`Create(litert::Environment & env, const std::string & model_filename, litert::HwAccelerators hardware_accelerators)`	`Expected< CompiledModel >` A simplified version of `Create` that uses default compilation options.
`Create(litert::Environment & env, BufferRef< uint8_t > model_buffer, litert::HwAccelerators hardware_accelerators)`	`Expected< CompiledModel >` An overload of `Create` that takes a buffer reference to the model instead of a filename.
`DefaultSignatureKey()`	`StringView` Returns the default signature key of the model.

Protected static functions
`BuildOptions(const Options & options, const internal::EnvironmentHolder & env)`	`Expected< internal::LiteRtOptionsPtr >`
`CheckCancelledWrapper(void *data)`	`bool`
`Create(litert::Environment & env, const LiteRtModel litert_model, Options & compilation_options)`	`Expected< CompiledModel >`
`Create(litert::Environment & env, const LiteRtModel litert_model, litert::HwAccelerators hardware_accelerators)`	`Expected< CompiledModel >`
`CreateBufferImpl(const Environment & env, const TensorBufferRequirements & buffer_requirements, const RankedTensorType & tensor_type)`	`Expected< TensorBuffer >` Creates a `TensorBuffer` with the given buffer requirements and tensor type.

Protected functions
`CompiledModel(internal::EnvironmentHolder & env, LiteRtModel litert_model, OwnHandle model_owned, LiteRtCompiledModel compiled_model, OwnHandle owned, internal::LiteRtOptionsPtr options)`	Constructs a `CompiledModel` instance.
`CreateInputOutputBuffer(size_t signature_index, StringView tensor_name, bool is_input) const`	`Expected< TensorBuffer >` Creates a `TensorBuffer` for a given signature index and tensor name.
`CreateInputOutputBuffer(StringView signature_name, StringView tensor_name, bool is_input) const`	`Expected< TensorBuffer >` Creates a `TensorBuffer` for a given signature and tensor name.
`CreateInputOutputBuffers(size_t signature_index, bool is_input) const`	`Expected< std::vector< TensorBuffer > >` Creates a vector of `TensorBuffer`s for a given signature subgraph.
`FindInputIndex(size_t signature_index, StringView input_name) const`	`Expected< size_t >` Returns the signature input index for a given input tensor name.
`FindOutputIndex(size_t signature_index, StringView output_name) const`	`Expected< size_t >` Returns the signature output index for a given output tensor name.
`GetEnvironment() const`	`Expected< Environment >`
`GetSignatureInputNamesImpl(size_t signature_index) const`	`Expected< std::vector< StringView > >`
`GetSignatureKeysImpl() const`	`Expected< std::vector< StringView > >`
`GetSignatureOutputNamesImpl(size_t signature_index) const`	`Expected< std::vector< StringView > >`
`RunCApiHelper(LiteRtParamIndex signature_index, size_t num_input_buffers, LiteRtTensorBuffer input_buffers, size_t num_output_buffers, LiteRtTensorBuffer output_buffers, bool & async) const`	`Expected< void >`
`RunCApiHelper(LiteRtParamIndex signature_index, size_t num_input_buffers, LiteRtTensorBuffer input_buffers, size_t num_output_buffers, LiteRtTensorBuffer output_buffers, bool & async, LiteRtOptions run_options) const`	`Expected< void >`
`RunCApiHelper(LiteRtParamIndex signature_index, size_t num_input_buffers, LiteRtTensorBuffer input_buffers, size_t num_output_buffers, LiteRtTensorBuffer output_buffers, bool & async, LiteRtOptions run_options, const LiteRtSchedulingInfo *scheduling_info) const`	`Expected< void >`
`RunHelper(size_t signature_index, Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers, bool & async) const`	`Expected< void >`
`RunHelper(size_t signature_index, Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers, bool & async, LiteRtOptions run_options) const`	`Expected< void >`
`RunHelper(size_t signature_index, Span< const TensorBuffer > input_buffers, Span< const TensorBuffer > output_buffers, bool & async, LiteRtOptions run_options, const LiteRtSchedulingInfo *scheduling_info) const`	`Expected< void >`
`RunMapHelper(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async) const`	`Expected< void >`
`RunMapHelper(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async, LiteRtOptions run_options) const`	`Expected< void >`
`RunMapHelper(StringView signature_key, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async, LiteRtOptions run_options, const LiteRtSchedulingInfo *scheduling_info) const`	`Expected< void >`
`RunMapWithIndexHelper(size_t signature_index, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async) const`	`Expected< void >`
`RunMapWithIndexHelper(size_t signature_index, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async, LiteRtOptions run_options) const`	`Expected< void >`
`RunMapWithIndexHelper(size_t signature_index, const FlatHashMap< StringView, TensorBuffer > & input_map, const FlatHashMap< StringView, TensorBuffer > & output_map, bool & async, LiteRtOptions run_options, const LiteRtSchedulingInfo *scheduling_info) const`	`Expected< void >`

Friend classes

::mediapipe::InferenceRunnerLiteRt

friend class ::mediapipe::InferenceRunnerLiteRt

benchmark::BenchmarkLiteRtModel

friend class benchmark::BenchmarkLiteRtModel

compiled_model_wrapper::CompiledModelWrapper

friend class compiled_model_wrapper::CompiledModelWrapper

lm::AudioLiteRtCompiledModelExecutor

friend class lm::AudioLiteRtCompiledModelExecutor

lm::EmbeddingLookupText

friend class lm::EmbeddingLookupText

lm::EndOfMultiModalEmbedding

friend class lm::EndOfMultiModalEmbedding

lm::FrontendModelWrapper

friend class lm::FrontendModelWrapper

lm::LlmLiteRtCompiledModelExecutorBase

friend class lm::LlmLiteRtCompiledModelExecutorBase

lm::LlmLiteRtCompiledModelExecutorDynamic

friend class lm::LlmLiteRtCompiledModelExecutorDynamic

lm::LlmLiteRtCompiledModelExecutorStatic

friend class lm::LlmLiteRtCompiledModelExecutorStatic

lm::LlmLiteRtMtpDrafter

friend class lm::LlmLiteRtMtpDrafter

lm::LlmLiteRtNpuCompiledModelExecutor

friend class lm::LlmLiteRtNpuCompiledModelExecutor

lm::VisionLiteRtCompiledModelExecutor

friend class lm::VisionLiteRtCompiledModelExecutor

Protected attributes

check_cancelled_func_

AnyInvocable< bool()> check_cancelled_func_

env_

internal::EnvironmentHolder env_

model_

internal::BaseHandle< LiteRtModel > model_

options_

internal::LiteRtOptionsPtr options_

Public functions

ClearErrors

Expected< void > ClearErrors() const

Clears all errors from the error reporter.

Note: This only works if the compiled model uses BufferErrorReporter, not StderrReporter.

ClearSchedulingInfo

Expected< void > ClearSchedulingInfo() const

Clears model-level default scheduling info.

CompiledModel

 CompiledModel()=default

CreateInputBuffer

Expected< TensorBuffer > CreateInputBuffer(
  StringView signature_name,
  StringView input_name
) const

Creates an input tensor buffer for the given signature and input name.

CreateInputBuffer

Expected< TensorBuffer > CreateInputBuffer(
  StringView input_name
) const

Creates an input tensor buffer for the default signature and a given input name.

CreateInputBuffers

Expected< std::vector< TensorBuffer > > CreateInputBuffers(
  StringView signature_name
) const

A helper function to create input tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the input tensor buffers.

CreateInputBuffers

Expected< std::vector< TensorBuffer > > CreateInputBuffers(
  size_t signature_index
) const

A helper function to create input tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the input tensor buffers.

CreateInputBuffers

Expected< std::vector< TensorBuffer > > CreateInputBuffers() const

A helper function to create input tensor buffers for the default signature.

It uses BufferRequirements and RankedTensorType to create the input tensor buffers.

CreateOutputBuffer

Expected< TensorBuffer > CreateOutputBuffer(
  StringView signature_name,
  StringView output_name
) const

Creates an output tensor buffer for the given signature and output name.

CreateOutputBuffer

Expected< TensorBuffer > CreateOutputBuffer(
  StringView output_name
) const

Creates an output tensor buffer for the default signature and a given output name.

CreateOutputBuffers

Expected< std::vector< TensorBuffer > > CreateOutputBuffers(
  StringView signature_name
) const

A helper function to create output tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the output tensor buffers.

CreateOutputBuffers

Expected< std::vector< TensorBuffer > > CreateOutputBuffers(
  size_t signature_index
) const

A helper function to create output tensor buffers for a given signature.

It uses BufferRequirements and RankedTensorType to create the output tensor buffers.

CreateOutputBuffers

Expected< std::vector< TensorBuffer > > CreateOutputBuffers() const

A helper function to create output tensor buffers for the default signature.

It uses BufferRequirements and RankedTensorType to create the output tensor buffers.

FindSignature

Expected< SimpleSignature > FindSignature(
  StringView signature_key
) const

GetDispatchAnnotation

Expected< std::optional< std::string > > GetDispatchAnnotation(
  size_t signature_index,
  StringView key
)

Gets a dispatch annotation from the compiled model.

Returns std::nullopt if the key does not exist.

GetDispatchAnnotation

Expected< std::optional< std::string > > GetDispatchAnnotation(
  StringView key
)

Overloaded version for the default signature (index 0).

GetDispatchAnnotation

Expected< std::optional< std::string > > GetDispatchAnnotation(
  StringView signature_name,
  StringView key
)

Overloaded version that takes a signature name instead of an index.

GetErrorMessages

Expected< std::string > GetErrorMessages() const

Gets all error messages from the error reporter as a single string.

Note: This only works if the compiled model uses BufferErrorReporter, not StderrReporter. The C++ wrapper automatically manages memory using RAII.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  StringView signature_name,
  StringView input_name
)

Gets input buffer requirements for the given signature and input name.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  size_t signature_index,
  size_t input_index
) const

Returns the buffer requirements for the n-th input tensor.

The returned TensorBufferRequirements is used to create the input tensor buffer.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  size_t signature_index,
  StringView input_name
) const

An overload of GetInputBufferRequirements that takes an input tensor name.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  size_t input_index
) const

Gets input buffer requirements of the default signature for the n-th input tensor.

GetInputBufferRequirements

Expected< TensorBufferRequirements > GetInputBufferRequirements(
  StringView input_name
) const

Gets input buffer requirements of the default signature for a given input name.

GetInputTensorLayout

Expected< Layout > GetInputTensorLayout(
  size_t signature_index,
  size_t input_index
) const

Returns the layout of the given input tensor.

This reflects the most recent shape requested via ResizeInputTensor or automatic resize during execution.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  size_t signature_index,
  size_t input_index
) const

Returns the tensor type for the n-th input tensor.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  size_t signature_index,
  StringView input_name
) const

Returns the tensor type for a given input tensor name.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  StringView signature_key,
  StringView input_name
) const

Returns the tensor type for a given input tensor name.

GetInputTensorType

Expected< RankedTensorType > GetInputTensorType(
  StringView input_name
) const

Gets the input tensor type of the default signature for a given input name.

GetNumSignatures

size_t GetNumSignatures() const

Returns the number of signatures defined in the model.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  StringView signature_name,
  StringView output_name
)

Gets output buffer requirements for the given signature and output name.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  size_t signature_index,
  size_t output_index
) const

Returns the buffer requirements for the given output tensor.

The returned TensorBufferRequirements is used to create the output tensor buffer.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  size_t signature_index,
  StringView output_name
) const

An overload of GetOutputBufferRequirements that takes an output tensor name.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  size_t output_index
) const

Gets input buffer requirements of the default signature for the n-th input tensor.

GetOutputBufferRequirements

Expected< TensorBufferRequirements > GetOutputBufferRequirements(
  StringView output_name
) const

Gets input buffer requirements of the default signature for a given input name.

GetOutputTensorLayouts

Expected< std::vector< Layout > > GetOutputTensorLayouts(
  size_t signature_index,
  bool update_allocation
) const

Returns the layouts of all output tensors for a given signature index.

If update_allocation is true, the tensor allocations will be updated to the current state of the compiled model.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  size_t signature_index,
  size_t output_index
) const

Returns the tensor type for the n-th output tensor.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  size_t signature_index,
  StringView output_name
) const

Returns the tensor type for a given output tensor name.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  StringView signature_key,
  StringView output_name
) const

Returns the tensor type for a given output tensor name.

GetOutputTensorType

Expected< RankedTensorType > GetOutputTensorType(
  StringView output_name
) const

Gets the output tensor type of the default signature for a given output name.

GetSignature

Expected< SimpleSignature > GetSignature(
  size_t signature_index
) const

Returns the signature at the given index.

GetSignatureIndex

Expected< size_t > GetSignatureIndex(
  StringView signature_key
) const

Returns the signature index for a given signature key.

Returns 0 if the signature key is empty.

GetSignatureInputNames

Expected< std::vector< StringView > > GetSignatureInputNames(
  size_t signature_index
) const

Returns the list of input names defined in the signature.

GetSignatureInputNames

Expected< std::vector< StringView > > GetSignatureInputNames() const

Returns the list of input names defined in the signature.

GetSignatureInputNames

Expected< std::vector< StringView > > GetSignatureInputNames(
  StringView signature_key
) const

Returns the list of input names defined in the signature.

GetSignatureKeys

Expected< std::vector< StringView > > GetSignatureKeys() const

Returns the list of signature key names defined in the signature.

GetSignatureOutputNames

Expected< std::vector< StringView > > GetSignatureOutputNames(
  size_t signature_index
) const

Returns the list of output names defined in the signature.

GetSignatureOutputNames

Expected< std::vector< StringView > > GetSignatureOutputNames() const

Returns the list of output names defined in the signature.

GetSignatureOutputNames

Expected< std::vector< StringView > > GetSignatureOutputNames(
  StringView signature_key
) const

Returns the list of output names defined in the signature.

GetSignatures

Expected< std::vector< SimpleSignature > > GetSignatures() const

Returns the list of signatures defined in the model.

IsFullyAccelerated

Expected< bool > IsFullyAccelerated()

Returns true if the compiled model is fully accelerated with the given hardware accelerators.

RemoveDispatchAnnotation

Expected< void > RemoveDispatchAnnotation(
  size_t signature_index,
  StringView key
)

Removes a dispatch annotation from the compiled model.

This succeeds even if the key does not exist.

RemoveDispatchAnnotation

Expected< void > RemoveDispatchAnnotation(
  StringView key
)

Overloaded version for the default signature (index 0).

RemoveDispatchAnnotation

Expected< void > RemoveDispatchAnnotation(
  StringView signature_name,
  StringView key
)

Overloaded version that takes a signature name instead of an index.

ReportError

Expected< void > ReportError(
  const char *format,
  Args &&... args
) const

Reports an error to the compiled model's error reporter.

Supports printf-style formatting for error messages.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  size_t signature_index,
  size_t input_index,
  Span< const int > dims
)

Resizes the specified input tensor to support dynamic shapes.

This function mirrors TFLite's ResizeInputTensorStrict API and requires the tensor signature to include dynamic dimensions. After calling this function, the compiled model will reallocate internal buffers as needed to accommodate the new tensor shape. For models that need relaxed validation, use ResizeInputTensorNonStrict.

Details

Parameters

`signature_index`	The index of the signature in the model.
`input_index`	The index of the input tensor in the signature.
`dims`	The new dimensions for the input tensor.

Returns

Success if the resize operation completes successfully, or an error with an appropriate status code on failure.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  size_t signature_index,
  StringView input_name,
  Span< const int > dims
)

Resizes the specified input tensor by name for the given signature.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  StringView signature_name,
  StringView input_name,
  Span< const int > dims
)

Resizes the specified input tensor by name for the given signature name.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  size_t input_index,
  Span< const int > dims
)

Resizes the specified input tensor of the default signature by index.

ResizeInputTensor

Expected< void > ResizeInputTensor(
  StringView input_name,
  Span< const int > dims
)

Resizes the specified input tensor of the default signature by name.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  size_t signature_index,
  size_t input_index,
  Span< const int > dims
)

Non-strict variants mirror TFLite's ResizeInputTensor behavior, allowing arbitrary shape updates when backends support them.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  size_t signature_index,
  StringView input_name,
  Span< const int > dims
)

Resizes the specified input tensor by name for the given signature.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  StringView signature_name,
  StringView input_name,
  Span< const int > dims
)

Resizes the specified input tensor by name for the given signature name.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  size_t input_index,
  Span< const int > dims
)

Resizes the specified input tensor of the default signature by index.

ResizeInputTensorNonStrict

Expected< void > ResizeInputTensorNonStrict(
  StringView input_name,
  Span< const int > dims
)

Resizes the specified input tensor of the default signature by name.

Run

Expected< void > Run(
  size_t signature_index,
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers
) const

Runs the model for a given signature index synchronously with the provided input/output TensorBuffers.

Run

Expected< void > Run(
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers
) const

Runs the model for the default signature synchronously with the provided input/output TensorBuffers.

Run

Expected< void > Run(
  size_t signature_index,
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers,
  Options *run_options
) const

Runs with per-run options for a given signature index.

Run

Expected< void > Run(
  size_t signature_index,
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs with per-request scheduling info for a given signature index.

Run

Expected< void > Run(
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers,
  Options *run_options
) const

Runs default signature with per-run options.

Run

Expected< void > Run(
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs default signature with per-request scheduling info.

Run

Expected< void > Run(
  StringView signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers
) const

Runs the model for a given signature key synchronously with the provided input/output TensorBuffers.

Run

Expected< void > Run(
  StringView signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  Options *run_options
) const

Runs by signature key with per-run options.

Run

Expected< void > Run(
  StringView signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs by signature key with per-request scheduling info.

Run

Expected< void > Run(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map
) const

Runs the model for a given signature key synchronously with the provided input/output TensorBuffer map.

If you have bound the input with external buffers through Options, you can skip providing those input buffers in the map.

Run

Expected< void > Run(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  Options *run_options
) const

Runs by signature key with per-run options using named maps.

Run

Expected< void > Run(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs by signature key with per-request scheduling info using named maps.

Run

Expected< void > Run(
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map
) const

Runs the model for the default signature synchronously with the provided input/output TensorBuffer map.

If you have bound the input with external buffers through Options, you can skip providing those input buffers in the map.

Run

Expected< void > Run(
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  Options *run_options
) const

Runs default signature with per-run options using named maps.

Run

Expected< void > Run(
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs default signature with per-request scheduling info using named maps.

RunAsync

Expected< void > RunAsync(
  size_t signature_index,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async
) const

Runs the model for a given signature index asynchronously, if possible, with the provided input/output TensorBuffers.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

RunAsync

Expected< void > RunAsync(
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async
) const

Runs the model for the default signature asynchronously, if possible, with the provided input/output TensorBuffers.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

RunAsync

Expected< void > RunAsync(
  size_t signature_index,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async,
  Options *run_options
) const

Runs asynchronously with per-run options for a given signature.

RunAsync

Expected< void > RunAsync(
  size_t signature_index,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs asynchronously with per-request scheduling info for a given signature.

RunAsync

Expected< void > RunAsync(
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async,
  Options *run_options
) const

Runs default signature asynchronously with per-run options.

RunAsync

Expected< void > RunAsync(
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs default signature asynchronously with per-request scheduling info.

RunAsync

Expected< void > RunAsync(
  StringView signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async
) const

Runs the model for a given signature key asynchronously, if possible, with the provided input/output TensorBuffers.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

RunAsync

Expected< void > RunAsync(
  StringView signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async,
  Options *run_options
) const

Runs by signature key asynchronously with per-run options.

RunAsync

Expected< void > RunAsync(
  StringView signature_key,
  const std::vector< TensorBuffer > & input_buffers,
  const std::vector< TensorBuffer > & output_buffers,
  bool & async,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs by signature key asynchronously with per-request scheduling info.

RunAsync

Expected< void > RunAsync(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async
) const

Runs the model for a given signature key asynchronously, if possible, with the provided input/output TensorBuffer map.

If asynchronous execution is possible, async will be set to true; otherwise, the function runs the model synchronously.

RunAsync

Expected< void > RunAsync(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async,
  Options *run_options
) const

Runs by signature key asynchronously with per-run options using named maps.

RunAsync

Expected< void > RunAsync(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async,
  const LiteRtSchedulingInfo & scheduling_info
) const

Runs by signature key asynchronously with per-request scheduling info using named maps.

SetCancellationFunction

void SetCancellationFunction(
  void *data,
  bool(*)(void *) check_cancelled_func
)

Sets a callback function that will be called after every node/op during model execution to check if the execution should be cancelled.

This behavior is defined here: tflite/core/subgraph.cc;l=1746-1750?q=tflite%20subgraph The callback should return true if execution should be cancelled.

Note: Use either this callback-based mechanism or the non-callback version (see below) with EnableCancellation/Cancel, but not both.

SetCancellationFunction

void SetCancellationFunction(
  AnyInvocable< bool()> check_cancelled_func
)

Sets a callback function for checking cancellation during execution.

The callback will be called periodically during model execution. This is a C++-friendly version of SetCancellationFunction.

SetDispatchAnnotation

Expected< void > SetDispatchAnnotation(
  size_t signature_index,
  StringView key,
  StringView value
)

Sets a dispatch annotation on the compiled model.

These annotations are propagated to dispatch graphs during model execution and provide runtime hints and metadata for hardware accelerator optimization.

SetDispatchAnnotation

Expected< void > SetDispatchAnnotation(
  StringView key,
  StringView value
)

Overloaded version for the default signature (index 0).

SetDispatchAnnotation

Expected< void > SetDispatchAnnotation(
  StringView signature_name,
  StringView key,
  StringView value
)

Overloaded version that takes a signature name instead of an index.

SetSchedulingInfo

Expected< void > SetSchedulingInfo(
  const LiteRtSchedulingInfo & scheduling_info
) const

Sets model-level default scheduling info.

Public static functions

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const std::string & model_filename,
  Options & compilation_options
)

Creates a CompiledModel from a TFLite file.

The model is loaded into memory, and the caller takes ownership of the returned CompiledModel object. The caller should keep the model alive until the CompiledModel is destroyed. The provided compilation_options are used for model compilation, and hardware_accelerators is used to select the accelerator, regardless of whether the model is AOT or JIT compiled.

Note: Even if the model is fully AOT-compiled for an NPU, you must specify the NPU accelerator in hardware_accelerators to use it properly.

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  BufferRef< uint8_t > model_buffer,
  Options & compilation_options
)

An overload of Create that takes a buffer reference to the model instead of a filename.

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const std::string & model_filename,
  litert::HwAccelerators hardware_accelerators
)

A simplified version of Create that uses default compilation options.

The provided hardware accelerator is used to select the target accelerator.

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  BufferRef< uint8_t > model_buffer,
  litert::HwAccelerators hardware_accelerators
)

An overload of Create that takes a buffer reference to the model instead of a filename.

DefaultSignatureKey

StringView DefaultSignatureKey()

Returns the default signature key of the model.

Protected static functions

BuildOptions

Expected< internal::LiteRtOptionsPtr > BuildOptions(
  const Options & options,
  const internal::EnvironmentHolder & env
)

CheckCancelledWrapper

bool CheckCancelledWrapper(
  void *data
)

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const LiteRtModel litert_model,
  Options & compilation_options
)

Create

Expected< CompiledModel > Create(
  litert::Environment & env,
  const LiteRtModel litert_model,
  litert::HwAccelerators hardware_accelerators
)

CreateBufferImpl

Expected< TensorBuffer > CreateBufferImpl(
  const Environment & env,
  const TensorBufferRequirements & buffer_requirements,
  const RankedTensorType & tensor_type
)

Creates a TensorBuffer with the given buffer requirements and tensor type.

Protected functions

CompiledModel

 CompiledModel(
  internal::EnvironmentHolder & env,
  LiteRtModel litert_model,
  OwnHandle model_owned,
  LiteRtCompiledModel compiled_model,
  OwnHandle owned,
  internal::LiteRtOptionsPtr options
)

Constructs a CompiledModel instance.

Details

Parameters

`model_owned`	Indicates whether the provided `litert_model` handle is owned by the `CompiledModel`.
`owned`	If `true`, the created object takes ownership of the `compiled_model` handle.

CreateInputOutputBuffer

Expected< TensorBuffer > CreateInputOutputBuffer(
  size_t signature_index,
  StringView tensor_name,
  bool is_input
) const

Creates a TensorBuffer for a given signature index and tensor name.

CreateInputOutputBuffer

Expected< TensorBuffer > CreateInputOutputBuffer(
  StringView signature_name,
  StringView tensor_name,
  bool is_input
) const

Creates a TensorBuffer for a given signature and tensor name.

CreateInputOutputBuffers

Expected< std::vector< TensorBuffer > > CreateInputOutputBuffers(
  size_t signature_index,
  bool is_input
) const

Creates a vector of TensorBuffers for a given signature subgraph.

FindInputIndex

Expected< size_t > FindInputIndex(
  size_t signature_index,
  StringView input_name
) const

Returns the signature input index for a given input tensor name.

FindOutputIndex

Expected< size_t > FindOutputIndex(
  size_t signature_index,
  StringView output_name
) const

Returns the signature output index for a given output tensor name.

GetEnvironment

Expected< Environment > GetEnvironment() const

GetSignatureInputNamesImpl

Expected< std::vector< StringView > > GetSignatureInputNamesImpl(
  size_t signature_index
) const

GetSignatureKeysImpl

Expected< std::vector< StringView > > GetSignatureKeysImpl() const

GetSignatureOutputNamesImpl

Expected< std::vector< StringView > > GetSignatureOutputNamesImpl(
  size_t signature_index
) const

RunCApiHelper

Expected< void > RunCApiHelper(
  LiteRtParamIndex signature_index,
  size_t num_input_buffers,
  LiteRtTensorBuffer *input_buffers,
  size_t num_output_buffers,
  LiteRtTensorBuffer *output_buffers,
  bool & async
) const

RunCApiHelper

Expected< void > RunCApiHelper(
  LiteRtParamIndex signature_index,
  size_t num_input_buffers,
  LiteRtTensorBuffer *input_buffers,
  size_t num_output_buffers,
  LiteRtTensorBuffer *output_buffers,
  bool & async,
  LiteRtOptions run_options
) const

RunCApiHelper

Expected< void > RunCApiHelper(
  LiteRtParamIndex signature_index,
  size_t num_input_buffers,
  LiteRtTensorBuffer *input_buffers,
  size_t num_output_buffers,
  LiteRtTensorBuffer *output_buffers,
  bool & async,
  LiteRtOptions run_options,
  const LiteRtSchedulingInfo *scheduling_info
) const

RunHelper

Expected< void > RunHelper(
  size_t signature_index,
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers,
  bool & async
) const

RunHelper

Expected< void > RunHelper(
  size_t signature_index,
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers,
  bool & async,
  LiteRtOptions run_options
) const

RunHelper

Expected< void > RunHelper(
  size_t signature_index,
  Span< const TensorBuffer > input_buffers,
  Span< const TensorBuffer > output_buffers,
  bool & async,
  LiteRtOptions run_options,
  const LiteRtSchedulingInfo *scheduling_info
) const

RunMapHelper

Expected< void > RunMapHelper(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async
) const

RunMapHelper

Expected< void > RunMapHelper(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async,
  LiteRtOptions run_options
) const

RunMapHelper

Expected< void > RunMapHelper(
  StringView signature_key,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async,
  LiteRtOptions run_options,
  const LiteRtSchedulingInfo *scheduling_info
) const

RunMapWithIndexHelper

Expected< void > RunMapWithIndexHelper(
  size_t signature_index,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async
) const

RunMapWithIndexHelper

Expected< void > RunMapWithIndexHelper(
  size_t signature_index,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async,
  LiteRtOptions run_options
) const

RunMapWithIndexHelper

Expected< void > RunMapWithIndexHelper(
  size_t signature_index,
  const FlatHashMap< StringView, TensorBuffer > & input_map,
  const FlatHashMap< StringView, TensorBuffer > & output_map,
  bool & async,
  LiteRtOptions run_options,
  const LiteRtSchedulingInfo *scheduling_info
) const

litert::CompiledModel

Summary

Inheritance

Constructors and Destructors

Friend classes

Protected attributes

Public functions

Public static functions

Protected static functions

Protected functions

Friend classes

::mediapipe::InferenceRunnerLiteRt

benchmark::BenchmarkLiteRtModel

compiled_model_wrapper::CompiledModelWrapper

lm::AudioLiteRtCompiledModelExecutor

lm::EmbeddingLookupText

lm::EndOfMultiModalEmbedding

lm::FrontendModelWrapper

lm::LlmLiteRtCompiledModelExecutorBase

lm::LlmLiteRtCompiledModelExecutorDynamic

lm::LlmLiteRtCompiledModelExecutorStatic

lm::LlmLiteRtMtpDrafter

lm::LlmLiteRtNpuCompiledModelExecutor

lm::VisionLiteRtCompiledModelExecutor

Protected attributes

check_cancelled_func_

env_

model_

options_

Public functions

ClearErrors

ClearSchedulingInfo

CompiledModel

CreateInputBuffer

CreateInputBuffer

CreateInputBuffers

CreateInputBuffers

CreateInputBuffers

CreateOutputBuffer

CreateOutputBuffer

CreateOutputBuffers

CreateOutputBuffers

CreateOutputBuffers

FindSignature

GetDispatchAnnotation

GetDispatchAnnotation

GetDispatchAnnotation

GetErrorMessages

GetInputBufferRequirements

GetInputBufferRequirements

GetInputBufferRequirements

GetInputBufferRequirements

GetInputBufferRequirements

GetInputTensorLayout

GetInputTensorType

GetInputTensorType

GetInputTensorType

GetInputTensorType

GetNumSignatures

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputBufferRequirements

GetOutputTensorLayouts

GetOutputTensorType

GetOutputTensorType

GetOutputTensorType

GetOutputTensorType

GetSignature

GetSignatureIndex

GetSignatureInputNames

GetSignatureInputNames

GetSignatureInputNames

GetSignatureKeys

GetSignatureOutputNames

GetSignatureOutputNames

GetSignatureOutputNames

GetSignatures

IsFullyAccelerated