Introducing Google AI Edge Portal: Benchmark Edge AI at scale. Sign-up to request access during private preview.

ObjectDetector

public final class ObjectDetector

Performs object detection on images.

The API expects a TFLite model with TFLite Model Metadata..

The API supports models with one image input tensor and four output tensors. To be more specific, here are the requirements.

Input image tensor (kTfLiteUInt8/kTfLiteFloat32)
- image input of size [batch x height x width x channels].
- batch inference is not supported (batch is required to be 1).
- only RGB inputs are supported (channels is required to be 3).
- if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.
Output tensors must be the 4 outputs of a DetectionPostProcess op, i.e:
- Location tensor (kTfLiteFloat32):
  - tensor of size [1 x num_results x 4], the inner array representing bounding boxes in the form [top, left, right, bottom].
  - BoundingBoxProperties are required to be attached to the metadata and must specify type=BOUNDARIES and coordinate_type=RATIO.
- Classes tensor (kTfLiteFloat32):
  - tensor of size [1 x num_results], each value representing the integer index of a class.
  - if label maps are attached to the metadata as TENSOR_VALUE_LABELS associated files, they are used to convert the tensor values into labels.
- scores tensor (kTfLiteFloat32):
  - tensor of size [1 x num_results], each value representing the score of the detected object.
- Number of detection tensor (kTfLiteFloat32):
  - integer num_results as a tensor of size [1].

An example of such model can be found on TensorFlow Hub..

Nested Classes

class ObjectDetector.ObjectDetectorOptions Options for setting up an ObjectDetector.

Public Methods

static ObjectDetector	createFromBuffer(Context context, ByteBuffer modelBuffer) Creates an `ObjectDetector` instance from a model buffer and the default `ObjectDetector.ObjectDetectorOptions`.
static ObjectDetector	createFromFile(Context context, String modelPath) Creates an `ObjectDetector` instance from a model file and the default `ObjectDetector.ObjectDetectorOptions`.
static ObjectDetector	createFromFile(Context context, File modelFile) Creates an `ObjectDetector` instance from a model file and the default `ObjectDetector.ObjectDetectorOptions`.
static ObjectDetector	createFromOptions(Context context, ObjectDetector.ObjectDetectorOptions detectorOptions) Creates an `ObjectDetector` instance from an `ObjectDetector.ObjectDetectorOptions`.
ObjectDetectorResult	detect(MPImage image) Performs object detection on the provided single image with default image processing options, i.e.
ObjectDetectorResult	detect(MPImage image, ImageProcessingOptions imageProcessingOptions) Performs object detection on the provided single image.
void	detectAsync(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Sends live image data to perform object detection, and the results will be available via the `OutputHandler.ResultListener` provided in the `ObjectDetector.ObjectDetectorOptions`.
void	detectAsync(MPImage image, long timestampMs) Sends live image data to perform object detection with default image processing options, i.e.
ObjectDetectorResult	detectForVideo(MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs) Performs object detection on the provided video frame.
ObjectDetectorResult	detectForVideo(MPImage image, long timestampMs) Performs object detection on the provided video frame with default image processing options, i.e.

Inherited Methods

From class com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi

void	close() Closes and cleans up the MediaPipe vision task.
static long	generateResultTimestampMs(RunningMode runningMode, Packet packet) Generates the timestamp of a vision task result object from the vision task running mode and the output packet.

From class java.lang.Object

boolean	equals(Object arg0)
final Class<?>	getClass()
int	hashCode()
final void	notify()
final void	notifyAll()
String	toString()
final void	wait(long arg0, int arg1)
final void	wait(long arg0)
final void	wait()

From interface java.lang.AutoCloseable

abstract void

close()

Public Methods

public static ObjectDetector createFromBuffer (Context context, ByteBuffer modelBuffer)

Creates an ObjectDetector instance from a model buffer and the default ObjectDetector.ObjectDetectorOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelBuffer	a direct `ByteBuffer` or a `ERROR(/MappedByteBuffer)` of the detection model.

Throws

	if there is an error during `ObjectDetector` creation.

public static ObjectDetector createFromFile (Context context, String modelPath)

Creates an ObjectDetector instance from a model file and the default ObjectDetector.ObjectDetectorOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelPath	path to the detection model with metadata in the assets.

Throws

	if there is an error during `ObjectDetector` creation.

public static ObjectDetector createFromFile (Context context, File modelFile)

Creates an ObjectDetector instance from a model file and the default ObjectDetector.ObjectDetectorOptions.

Parameters

context	an Android `ERROR(/Context)`.
modelFile	the detection model `File` instance.

Throws

IOException	if an I/O error occurs when opening the tflite model file.
	if there is an error during `ObjectDetector` creation.

public static ObjectDetector createFromOptions (Context context, ObjectDetector.ObjectDetectorOptions detectorOptions)

Creates an ObjectDetector instance from an ObjectDetector.ObjectDetectorOptions.

Parameters

context	an Android `ERROR(/Context)`.
detectorOptions	an `ObjectDetector.ObjectDetectorOptions` instance.

Throws

	if there is an error during `ObjectDetector` creation.

public ObjectDetectorResult detect (MPImage image)

Performs object detection on the provided single image with default image processing options, i.e. without any rotation applied. Only use this method when the ObjectDetector is created with ERROR(/RunningMode.IMAGE).

ObjectDetector supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.

Throws

	if there is an internal error.

public ObjectDetectorResult detect (MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs object detection on the provided single image. Only use this method when the ObjectDetector is created with ERROR(/RunningMode.IMAGE).

ObjectDetector supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying `ImageProcessingOptions.regionOfInterest()` will result in this method throwing an IllegalArgumentException.

Throws

IllegalArgumentException	if the `ImageProcessingOptions` specify a region-of-interest.
	if there is an internal error.

public void detectAsync (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Sends live image data to perform object detection, and the results will be available via the OutputHandler.ResultListener provided in the ObjectDetector.ObjectDetectorOptions. Only use this method when the ObjectDetector is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying `ImageProcessingOptions.regionOfInterest()` will result in this method throwing an IllegalArgumentException.
timestampMs	the input timestamp (in milliseconds).

Throws

IllegalArgumentException	if the `ImageProcessingOptions` specify a region-of-interest.
	if there is an internal error.

public void detectAsync (MPImage image, long timestampMs)

Sends live image data to perform object detection with default image processing options, i.e. without any rotation applied, and the results will be available via the OutputHandler.ResultListener provided in the ObjectDetector.ObjectDetectorOptions. Only use this method when the ObjectDetector is created with ERROR(/RunningMode.LIVE_STREAM).

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the object detector. The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.

public ObjectDetectorResult detectForVideo (MPImage image, ImageProcessingOptions imageProcessingOptions, long timestampMs)

Performs object detection on the provided video frame. Only use this method when the ObjectDetector is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
imageProcessingOptions	the `ImageProcessingOptions` specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying `ImageProcessingOptions.regionOfInterest()` will result in this method throwing an IllegalArgumentException.
timestampMs	the input timestamp (in milliseconds).

Throws

IllegalArgumentException	if the `ImageProcessingOptions` specify a region-of-interest.
	if there is an internal error.

public ObjectDetectorResult detectForVideo (MPImage image, long timestampMs)

Performs object detection on the provided video frame with default image processing options, i.e. without any rotation applied. Only use this method when the ObjectDetector is created with ERROR(/RunningMode.VIDEO).

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

ObjectDetector supports the following color space types:

ERROR(/Bitmap.Config.ARGB_8888)

Parameters

image	a MediaPipe `MPImage` object for processing.
timestampMs	the input timestamp (in milliseconds).

Throws

	if there is an internal error.