Introducing Google AI Edge Portal: Benchmark Edge AI at scale. Sign-up to request access during private preview.

ImageEmbedder

public final class ImageEmbedder

Performs embedding extraction on images.

The API expects a TFLite model with optional, but strongly recommended, TFLite Model Metadata..

The API supports models with one image input tensor and one or more output tensors. To be more specific, here are the requirements.

Input image tensor (kTfLiteUInt8/kTfLiteFloat32)
- image input of size [batch x height x width x channels].
- batch inference is not supported (batch is required to be 1).
- only RGB inputs are supported (channels is required to be 3).
- if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.
At least one output tensor (kTfLiteUInt8/kTfLiteFloat32) with shape [1 x N] where N is the number of dimensions in the produced embeddings.

Nested Classes

class ImageEmbedder.ImageEmbedderOptions Options for setting up and ImageEmbedder.

Public Methods

static double	cosineSimilarity(Embedding u, Embedding v) Utility function to compute cosine similarity between two `Embedding` objects.
static ImageEmbedder	createFromBuffer(Context context, ByteBuffer modelBuffer) Creates an `ImageEmbedder` instance from a model buffer and default `ImageEmbedder.ImageEmbedderOptions`.
static ImageEmbedder	createFromFile(Context context, String modelPath) Creates an `ImageEmbedder` instance from a model file and default `ImageEmbedder.ImageEmbedderOptions`.
static ImageEmbedder	createFromFile(Context context, File modelFile) Creates an `ImageEmbedder` instance from a model file and default `ImageEmbedder.ImageEmbedderOptions`.
static ImageEmbedder	createFromOptions(Context context, ImageEmbedder.ImageEmbedderOptions options) Creates an `ImageEmbedder` instance from an `ImageEmbedder.ImageEmbedderOptions` instance.
ImageEmbedderResult	embed(MPImage image, ImageProcessingOptions imageProcessingOptions) Performs embedding extraction on the provided single image.
ImageEmbedderResult	embed(MPImage image) Performs embedding extraction on the provided single image with default image processing options, i.e.
void	embedAsync(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Sends live image data to perform embedding extraction, and the results will be available via the `OutputHandler.ResultListener` provided in the `ImageEmbedder.ImageEmbedderOptions`.
void	embedAsync(MPImage image, long timestampMs) Sends live image data to perform embedding extraction with default image processing options, i.e.
ImageEmbedderResult	embedForVideo(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Performs embedding extraction on the provided video frame.
ImageEmbedderResult	embedForVideo(MPImage image, long timestampMs) Performs embedding extraction on the provided video frame with default image processing options, i.e.

Inherited Methods

From class com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi

void	close() Closes and cleans up the MediaPipe vision task.
static long	generateResultTimestampMs(RunningMode runningMode, Packet packet) Generates the timestamp of a vision task result object from the vision task running mode and the output packet.

From class java.lang.Object

boolean	equals(Object arg0)
final Class<?>	getClass()
int	hashCode()
final void	notify()
final void	notifyAll()
String	toString()
final void	wait(long arg0, int arg1)
final void	wait(long arg0)
final void	wait()

From interface java.lang.AutoCloseable

abstract void

close()

Public Methods

public static double cosineSimilarity (Embedding u, Embedding v)

Utility function to compute cosine similarity between two Embedding objects.

Parameters

u
v

Throws

IllegalArgumentException	if the embeddings are of different types (float vs. quantized), have different sizes, or have an L2-norm of 0.

public static ImageEmbedder createFromBuffer (Context context, ByteBuffer modelBuffer)

Creates an ImageEmbedder instance from a model buffer and default ImageEmbedder.ImageEmbedderOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelBuffer	a direct `ByteBuffer` or a `ERROR(/MappedByteBuffer)` of the embedding model.

Throws

	if there is an error during `ImageEmbedder` creation.

public static ImageEmbedder createFromFile (Context context, String modelPath)

Creates an ImageEmbedder instance from a model file and default ImageEmbedder.ImageEmbedderOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelPath	path to the embedding model in the assets.

Throws

	if there is an error during `ImageEmbedder` creation.

public static ImageEmbedder createFromFile (Context context, File modelFile)

Creates an ImageEmbedder instance from a model file and default ImageEmbedder.ImageEmbedderOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelFile	the embedding model `File` instance.

Throws

IOException	if an I/O error occurs when opening the tflite model file.
	if there is an error during `ImageEmbedder` creation.

public static ImageEmbedder createFromOptions (Context context, ImageEmbedder.ImageEmbedderOptions options)

Creates an ImageEmbedder instance from an ImageEmbedder.ImageEmbedderOptions instance.

Parameters

context	an Android `ERROR(/Context)`.
options	an `ImageEmbedder.ImageEmbedderOptions` instance.

Throws

	if there is an error during `ImageEmbedder` creation.

public ImageEmbedderResult embed (MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs embedding extraction on the provided single image. Only use this method when the ImageEmbedder is created with ERROR(/RunningMode.IMAGE).

ImageEmbedder supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference.

Throws

	if there is an internal error.

public ImageEmbedderResult embed (MPImage image)

Performs embedding extraction on the provided single image with default image processing options, i.e. using the whole image as region-of-interest and without any rotation applied. Only use this method when the ImageEmbedder is created with ERROR(/RunningMode.IMAGE).

ImageEmbedder supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.

Throws

	if there is an internal error.

public void embedAsync (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Sends live image data to perform embedding extraction, and the results will be available via the OutputHandler.ResultListener provided in the ImageEmbedder.ImageEmbedderOptions. Only use this method when the ImageEmbedder is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ImageEmbedder supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public void embedAsync (MPImage image, long timestampMs)

Sends live image data to perform embedding extraction with default image processing options, i.e. using the whole image as region-of-interest and without any rotation applied, and the results will be available via the OutputHandler.ResultListener provided in the ImageEmbedder.ImageEmbedderOptions. Only use this method when the ImageEmbedder is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ImageEmbedder supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public ImageEmbedderResult embedForVideo (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Performs embedding extraction on the provided video frame. Only use this method when the ImageEmbedder is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageEmbedder supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public ImageEmbedderResult embedForVideo (MPImage image, long timestampMs)

Performs embedding extraction on the provided video frame with default image processing options, i.e. using the whole image as region-of-interest and without any rotation applied. Only use this method when the ImageEmbedder is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ImageEmbedder supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.