Introducing LiteRT: Google's high-performance runtime for on-device AI, formerly known as TensorFlow Lite. Learn more

GestureRecognizer

public final class GestureRecognizer

Performs gesture recognition on images.

This API expects a pre-trained hand gesture model asset bundle, or a custom one created using Model Maker. See .

Input image MPImage
- The image that gesture recognition runs on.
Output GestureRecognizerResult GestureRecognizerResult
- A GestureRecognizerResult containing hand landmarks and recognized hand gestures.

Nested Classes

class GestureRecognizer.GestureRecognizerOptions Options for setting up an GestureRecognizer.

Public Methods

static GestureRecognizer	createFromBuffer(Context context, ByteBuffer modelBuffer) Creates a `GestureRecognizer` instance from a model buffer and the default `GestureRecognizer.GestureRecognizerOptions`.
static GestureRecognizer	createFromFile(Context context, String modelPath) Creates a `GestureRecognizer` instance from a model file and the default `GestureRecognizer.GestureRecognizerOptions`.
static GestureRecognizer	createFromFile(Context context, File modelFile) Creates a `GestureRecognizer` instance from a model file and the default `GestureRecognizer.GestureRecognizerOptions`.
static GestureRecognizer	createFromOptions(Context context, GestureRecognizer.GestureRecognizerOptions recognizerOptions) Creates a `GestureRecognizer` instance from a `GestureRecognizer.GestureRecognizerOptions`.
GestureRecognizerResult	recognize(MPImage image) Performs gesture recognition on the provided single image with default image processing options, i.e.
GestureRecognizerResult	recognize(MPImage image, ImageProcessingOptions imageProcessingOptions) Performs gesture recognition on the provided single image.
void	recognizeAsync(MPImage image, long timestampMs) Sends live image data to perform gesture recognition with default image processing options, i.e.
void	recognizeAsync(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Sends live image data to perform gesture recognition, and the results will be available via the `OutputHandler.ResultListener` provided in the `GestureRecognizer.GestureRecognizerOptions`.
GestureRecognizerResult	recognizeForVideo(MPImage image, long timestampMs) Performs gesture recognition on the provided video frame with default image processing options, i.e.
GestureRecognizerResult	recognizeForVideo(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Performs gesture recognition on the provided video frame.

Inherited Methods

From class com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi

void	close() Closes and cleans up the MediaPipe vision task.
static long	generateResultTimestampMs(RunningMode runningMode, Packet packet) Generates the timestamp of a vision task result object from the vision task running mode and the output packet.

From class java.lang.Object

boolean	equals(Object arg0)
final Class<?>	getClass()
int	hashCode()
final void	notify()
final void	notifyAll()
String	toString()
final void	wait(long arg0, int arg1)
final void	wait(long arg0)
final void	wait()

From interface java.lang.AutoCloseable

abstract void

close()

Public Methods

public static GestureRecognizer createFromBuffer (Context context, ByteBuffer modelBuffer)

Creates a GestureRecognizer instance from a model buffer and the default GestureRecognizer.GestureRecognizerOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelBuffer	a direct `ByteBuffer` or a `ERROR(/MappedByteBuffer)` of the detection model.

Throws

	if there is an error during `GestureRecognizer` creation.

public static GestureRecognizer createFromFile (Context context, String modelPath)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizer.GestureRecognizerOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelPath	path to the gesture recognition model with metadata in the assets.

Throws

	if there is an error during `GestureRecognizer` creation.

public static GestureRecognizer createFromFile (Context context, File modelFile)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizer.GestureRecognizerOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelFile	the gesture recognition model `File` instance.

Throws

IOException	if an I/O error occurs when opening the tflite model file.
	if there is an error during `GestureRecognizer` creation.

public static GestureRecognizer createFromOptions (Context context, GestureRecognizer.GestureRecognizerOptions recognizerOptions)

Creates a GestureRecognizer instance from a GestureRecognizer.GestureRecognizerOptions.

Parameters

context	an Android `ERROR(/Context)`.
recognizerOptions	a `GestureRecognizer.GestureRecognizerOptions` instance.

Throws

	if there is an error during `GestureRecognizer` creation.

public GestureRecognizerResult recognize (MPImage image)

Performs gesture recognition on the provided single image with default image processing options, i.e. without any rotation applied. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.IMAGE). TODO update java doc for input image format.

GestureRecognizer supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.

Throws

	if there is an internal error.

public GestureRecognizerResult recognize (MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs gesture recognition on the provided single image. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.IMAGE). TODO update java doc for input image format.

GestureRecognizer supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying `ImageProcessingOptions.regionOfInterest()` will result in this method throwing an IllegalArgumentException.

Throws

IllegalArgumentException	if the `ImageProcessingOptions` specify a region-of-interest.
	if there is an internal error.

public void recognizeAsync (MPImage image, long timestampMs)

Sends live image data to perform gesture recognition with default image processing options, i.e. without any rotation applied, and the results will be available via the OutputHandler.ResultListener provided in the GestureRecognizer.GestureRecognizerOptions. Only use this method when the ERROR(/GestureRecognition) is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the gesture recognizer. The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public void recognizeAsync (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Sends live image data to perform gesture recognition, and the results will be available via the OutputHandler.ResultListener provided in the GestureRecognizer.GestureRecognizerOptions. Only use this method when the ERROR(/GestureRecognition) is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the gesture recognizer. The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying `ImageProcessingOptions.regionOfInterest()` will result in this method throwing an IllegalArgumentException.
timestampMs	the input timestamp (in milliseconds).

Throws

IllegalArgumentException	if the `ImageProcessingOptions` specify a region-of-interest.
	if there is an internal error.

public GestureRecognizerResult recognizeForVideo (MPImage image, long timestampMs)

Performs gesture recognition on the provided video frame with default image processing options, i.e. without any rotation applied. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public GestureRecognizerResult recognizeForVideo (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Performs gesture recognition on the provided video frame. Only use this method when the GestureRecognizer is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying `ImageProcessingOptions.regionOfInterest()` will result in this method throwing an IllegalArgumentException.
timestampMs	the input timestamp (in milliseconds).

Throws

IllegalArgumentException	if the `ImageProcessingOptions` specify a region-of-interest.
	if there is an internal error.