litert::TensorBuffer

#include <litert_tensor_buffer.h>

A C++ wrapper for LiteRtTensorBuffer, representing a tensor and its associated backing buffer.

Summary

Inheritance

Inherits from: internal::BaseHandle< LiteRtTensorBuffer >

Constructors and Destructors
TensorBuffer()

Public types
LockMode enum

Public functions
BufferType() const
Expected< TensorBufferType >
Clear()
Expected< void >
Clears the tensor buffer possibly asynchronously.
ClearEvent()
Expected< void >
Duplicate() const
Creates a duplicate of the current TensorBuffer object.
GetAhwb() const
litert::Expected< AHardwareBuffer * >
GetDmaBuf() const
GetEvent() const
GetGlBuffer() const
GetGlTexture() const
GetMetalBuffer() const
Expected< void * >
GetOpenClMemory() const
Expected< LiteRtClMem >
GetVulkanMemory() const
Expected< HwMemoryHandle >
GetWebGpuBuffer() const
Expected< HwMemoryHandle >
HasEvent() const
bool
HasType(const RankedTensorType & type) const
bool
HasType(const LiteRtRankedTensorType & type) const
bool
IsMetalMemory() const
bool
Returns true if the tensor buffer is a Metal memory object.
IsOpenClMemory() const
bool
Returns true if the tensor buffer is an OpenCL memory object.
IsVulkanMemory() const
bool
Returns true if the tensor buffer is a Vulkan memory object.
IsWebGpuMemory() const
bool
Returns true if the tensor buffer is a WebGPU memory object.
Lock(LockMode mode)
Expected< void * >
Locks the tensor buffer and returns a pointer to the host memory.
Offset() const
Expected< size_t >
PackedSize() const
Expected< size_t >
Returns the size of the tensor buffer in packed bytes.
Read(Span< T > data)
Expected< void >
Reads data from the tensor buffer into a user-provided Span.
SetEvent(Event && event)
Expected< void >
Sets the C++ Event object for the tensor buffer.
Size() const
Expected< size_t >
Returns the size of the underlying hardware tensor buffer.
TensorType() const
Expected< RankedTensorType >
Unlock()
Expected< void >
Unlocks the tensor buffer and may synchronize the underlying hardware tensor buffer.
Write(Span< const T > data)
Expected< void >
Writes data from a user-provided Span to the tensor buffer.

Public static functions
CreateFromAhwb(const Environment & env, const RankedTensorType & tensor_type, AHardwareBuffer *ahwb, size_t ahwb_offset)
Creates a TensorBuffer that wraps an Android Hardware Buffer.
CreateFromClBuffer(const Environment & env, const RankedTensorType & tensor_type, TensorBufferType buffer_type, LiteRtClMem cl_memory, size_t size_bytes)
CreateFromGlBuffer(const Environment & env, const RankedTensorType & tensor_type, LiteRtGLenum target, LiteRtGLuint id, size_t size_bytes, size_t offset)
CreateFromGlTexture(const Environment & env, const RankedTensorType & tensor_type, LiteRtGLenum target, LiteRtGLuint id, LiteRtGLenum format, size_t size_bytes, LiteRtGLint layer)
CreateFromHostMemory(const Environment & env, const RankedTensorType & tensor_type, void *host_mem_addr, size_t buffer_size)
Creates a TensorBuffer that wraps the provided host memory.
CreateFromHostMemory(const RankedTensorType & tensor_type, void *host_mem_addr, size_t buffer_size) Deprecated. Use API with explicit Environment instead.
CreateManaged(const Environment & env, TensorBufferType buffer_type, const RankedTensorType & tensor_type, size_t buffer_size)
Creates a managed TensorBuffer of a given buffer type and size.
CreateManagedFromRequirements(const Environment & env, const RankedTensorType & tensor_type, const TensorBufferRequirements & requirements)
Creates a managed TensorBuffer from requirements.
CreateManagedHostMemory(const RankedTensorType & tensor_type, size_t buffer_size) Deprecated. Use API with explicit Environment instead.
Creates a managed host memory TensorBuffer using the default environment (if applicable).
ToLiteRtLockMode(LockMode mode)
LiteRtTensorBufferLockMode
WrapCObject(LiteRtTensorBuffer tensor_buffer, OwnHandle owned)
WrapCObject(const internal::EnvironmentHolder & env, LiteRtTensorBuffer tensor_buffer, OwnHandle owned)

Structs
litert::TensorBuffer::DmaBuf
litert::TensorBuffer::GlBuffer
litert::TensorBuffer::GlTexture

Public types

LockMode

 LockMode

Public functions

BufferType

Expected< TensorBufferType > BufferType() const

Clear

Expected< void > Clear()

Clears the tensor buffer possibly asynchronously.

It may return immediately after scheduling a clear operation though it guarantees that Read() will return data cleared, i.e. all zeros.

ClearEvent

Expected< void > ClearEvent()

Duplicate

Expected< TensorBuffer > Duplicate() const

Creates a duplicate of the current TensorBuffer object.

The returned object is reference-counted, so the underlying LiteRtTensorBuffer handle is not released until the last reference is removed.

GetAhwb

litert::Expected< AHardwareBuffer * > GetAhwb() const

GetDmaBuf

litert::Expected< DmaBuf > GetDmaBuf() const

GetEvent

Expected< Event > GetEvent() const

GetGlBuffer

Expected< GlBuffer > GetGlBuffer() const

GetGlTexture

Expected< GlTexture > GetGlTexture() const

GetMetalBuffer

Expected< void * > GetMetalBuffer() const

GetOpenClMemory

Expected< LiteRtClMem > GetOpenClMemory() const

GetVulkanMemory

Expected< HwMemoryHandle > GetVulkanMemory() const

GetWebGpuBuffer

Expected< HwMemoryHandle > GetWebGpuBuffer() const

HasEvent

bool HasEvent() const

HasType

bool HasType(
  const RankedTensorType & type
) const

HasType

bool HasType(
  const LiteRtRankedTensorType & type
) const

IsMetalMemory

bool IsMetalMemory() const

Returns true if the tensor buffer is a Metal memory object.

IsOpenClMemory

bool IsOpenClMemory() const

Returns true if the tensor buffer is an OpenCL memory object.

IsVulkanMemory

bool IsVulkanMemory() const

Returns true if the tensor buffer is a Vulkan memory object.

IsWebGpuMemory

bool IsWebGpuMemory() const

Returns true if the tensor buffer is a WebGPU memory object.

Lock

Expected< void * > Lock(
  LockMode mode
)

Locks the tensor buffer and returns a pointer to the host memory.

The mapped host memory is synchronized with the underlying hardware tensor buffer. For some memory types like AHWB, the pointer reflects changes of underlying memory after locking, but it's not always the case. For example, for most GPU memory types, the pointer memory is a snapshot of the underlying memory at the time of locking. If the underlying memory is modified after locking, the changes are not reflected in the mapped host memory.

The same thing happens to write lock. If you locked a tensor buffer for write, the underlying memory is not guaranteed to be updated immediately. For most GPU memory types, the CPU -> GPU copy is only performed when the tensor buffer is unlocked.

Offset

Expected< size_t > Offset() const

PackedSize

Expected< size_t > PackedSize() const

Returns the size of the tensor buffer in packed bytes.

This size is used for reading/writing data on a locked tensor buffer.

Read

Expected< void > Read(
  Span< T > data
)

Reads data from the tensor buffer into a user-provided Span.

If the provided buffer is smaller than the tensor buffer, data will be read up to the size of the provided buffer. Returns an error if the provided buffer is larger than the tensor buffer.

SetEvent

Expected< void > SetEvent(
  Event && event
)

Sets the C++ Event object for the tensor buffer.

This function takes ownership of the provided Event object.

Size

Expected< size_t > Size() const

Returns the size of the underlying hardware tensor buffer.

This size can differ from PackedSize() if striding and padding exist.

TensorBuffer

 TensorBuffer()

TensorType

Expected< RankedTensorType > TensorType() const

Unlock

Expected< void > Unlock()

Unlocks the tensor buffer and may synchronize the underlying hardware tensor buffer.

For most GPU memory types, the CPU -> GPU copy is performed if it was locked for write.

Write

Expected< void > Write(
  Span< const T > data
)

Writes data from a user-provided Span to the tensor buffer.

Returns an error if the provided buffer is larger than the tensor buffer's size.

Public static functions

CreateFromAhwb

Expected< TensorBuffer > CreateFromAhwb(
  const Environment & env,
  const RankedTensorType & tensor_type,
  AHardwareBuffer *ahwb,
  size_t ahwb_offset
)

Creates a TensorBuffer that wraps an Android Hardware Buffer.

The provided AHardwareBuffer is not owned by the TensorBuffer and must outlive it.

Details
Parameters
ahwb_offset
The offset in bytes from the start of the AHardwareBuffer where the tensor data begins.

CreateFromClBuffer

Expected< TensorBuffer > CreateFromClBuffer(
  const Environment & env,
  const RankedTensorType & tensor_type,
  TensorBufferType buffer_type,
  LiteRtClMem cl_memory,
  size_t size_bytes
)

CreateFromGlBuffer

Expected< TensorBuffer > CreateFromGlBuffer(
  const Environment & env,
  const RankedTensorType & tensor_type,
  LiteRtGLenum target,
  LiteRtGLuint id,
  size_t size_bytes,
  size_t offset
)

CreateFromGlTexture

Expected< TensorBuffer > CreateFromGlTexture(
  const Environment & env,
  const RankedTensorType & tensor_type,
  LiteRtGLenum target,
  LiteRtGLuint id,
  LiteRtGLenum format,
  size_t size_bytes,
  LiteRtGLint layer
)

CreateFromHostMemory

Expected< TensorBuffer > CreateFromHostMemory(
  const Environment & env,
  const RankedTensorType & tensor_type,
  void *host_mem_addr,
  size_t buffer_size
)

Creates a TensorBuffer that wraps the provided host memory.

The provided host memory is not owned by the TensorBuffer object and must outlive it. Callers are responsible for ensuring that the pointer is aligned to at least LITERT_HOST_MEMORY_BUFFER_ALIGNMENT bytes and that any required padding for delegates like XNNPACK is included and initialized.

CreateFromHostMemory

Expected< TensorBuffer > CreateFromHostMemory(
  const RankedTensorType & tensor_type,
  void *host_mem_addr,
  size_t buffer_size
)

Deprecated. Use API with explicit Environment instead.

CreateManaged

Expected< TensorBuffer > CreateManaged(
  const Environment & env,
  TensorBufferType buffer_type,
  const RankedTensorType & tensor_type,
  size_t buffer_size
)

Creates a managed TensorBuffer of a given buffer type and size.

The returned object is owned by the caller. For host memory, this allocator guarantees LITERT_HOST_MEMORY_BUFFER_ALIGNMENT alignment and reserves any delegate-specific padding (e.g., XNNPACK extra bytes), so callers do not need to over-allocate manually.

CreateManagedFromRequirements

Expected< TensorBuffer > CreateManagedFromRequirements(
  const Environment & env,
  const RankedTensorType & tensor_type,
  const TensorBufferRequirements & requirements
)

Creates a managed TensorBuffer from requirements.

The returned object is owned by the caller. It automatically selects the best buffer type and applies required alignment and padding.

CreateManagedHostMemory

Expected< TensorBuffer > CreateManagedHostMemory(
  const RankedTensorType & tensor_type,
  size_t buffer_size
)

Creates a managed host memory TensorBuffer using the default environment (if applicable).

The returned object is owned by the caller. Deprecated. Use API with explicit Environment instead.

ToLiteRtLockMode

LiteRtTensorBufferLockMode ToLiteRtLockMode(
  LockMode mode
)

WrapCObject

TensorBuffer WrapCObject(
  LiteRtTensorBuffer tensor_buffer,
  OwnHandle owned
)

WrapCObject

TensorBuffer WrapCObject(
  const internal::EnvironmentHolder & env,
  LiteRtTensorBuffer tensor_buffer,
  OwnHandle owned
)