Performs interactive segmentation on images.
Users can represent user interaction through RegionOfInterest, which gives a hint to InteractiveSegmenter to perform segmentation focusing on the given region of interest.
The API expects a TFLite model with mandatory TFLite Model Metadata.
Input tensor: (kTfLiteUInt8/kTfLiteFloat32) - image input of size [batch x height x width x channels]. - batch inference is not supported (batch is required to be 1). - RGB inputs is supported (channels is required to be 3). - if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization. Output tensors: (kTfLiteUInt8/kTfLiteFloat32) - list of segmented masks. - if output_type is CATEGORY_MASK, uint8 Image, Image vector of size 1. - if output_type is CONFIDENCE_MASK, float32 Image list of size channels. - batch is always 1
Signature:
export declare class InteractiveSegmenter extends VisionTaskRunner
Extends: VisionTaskRunner
Methods
| Method | Modifiers | Description |
|---|---|---|
| createFromModelBuffer(wasmFileset, modelAssetBuffer) | static |
Initializes the Wasm runtime and creates a new interactive segmenter based on the provided model asset buffer. |
| createFromModelPath(wasmFileset, modelAssetPath) | static |
Initializes the Wasm runtime and creates a new interactive segmenter based on the path to the model asset. |
| createFromOptions(wasmFileset, interactiveSegmenterOptions) | static |
Initializes the Wasm runtime and creates a new interactive segmenter from the provided options. |
| segment(image, roi, callback) | Performs interactive segmentation on the provided single image and invokes the callback with the response. The method returns synchronously once the callback returns. The roi parameter is used to represent a user's region of interest for segmentation. |
|
| segment(image, roi, imageProcessingOptions, callback) | Performs interactive segmentation on the provided single image and invokes the callback with the response. The method returns synchronously once the callback returns. The roi parameter is used to represent a user's region of interest for segmentation.The 'imageProcessingOptions' parameter can be used to specify the rotation to apply to the image before performing segmentation, by setting its 'rotationDegrees' field. Note that specifying a region-of-interest using the 'regionOfInterest' field is NOT supported and will result in an error. |
|
| segment(image, roi) | Performs interactive segmentation on the provided video frame and returns the segmentation result. This method creates a copy of the resulting masks and should not be used in high-throughput applications. The roi parameter is used to represent a user's region of interest for segmentation. |
|
| segment(image, roi, imageProcessingOptions) | Performs interactive segmentation on the provided video frame and returns the segmentation result. This method creates a copy of the resulting masks and should not be used in high-throughput applications. The roi parameter is used to represent a user's region of interest for segmentation.The 'imageProcessingOptions' parameter can be used to specify the rotation to apply to the image before performing segmentation, by setting its 'rotationDegrees' field. Note that specifying a region-of-interest using the 'regionOfInterest' field is NOT supported and will result in an error. |
|
| setOptions(options) | Sets new options for the interactive segmenter.Calling setOptions() with a subset of options only affects those options. You can reset an option back to its default value by explicitly setting it to undefined. |
InteractiveSegmenter.createFromModelBuffer()
Initializes the Wasm runtime and creates a new interactive segmenter based on the provided model asset buffer.
Signature:
static createFromModelBuffer(wasmFileset: WasmFileset, modelAssetBuffer: Uint8Array): Promise<InteractiveSegmenter>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| wasmFileset | WasmFileset | A configuration object that provides the location of the Wasm binary and its loader. |
| modelAssetBuffer | Uint8Array | A binary representation of the model. A new InteractiveSegmenter. |
Returns:
Promise<InteractiveSegmenter>
InteractiveSegmenter.createFromModelPath()
Initializes the Wasm runtime and creates a new interactive segmenter based on the path to the model asset.
Signature:
static createFromModelPath(wasmFileset: WasmFileset, modelAssetPath: string): Promise<InteractiveSegmenter>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| wasmFileset | WasmFileset | A configuration object that provides the location of the Wasm binary and its loader. |
| modelAssetPath | string | The path to the model asset. A new InteractiveSegmenter. |
Returns:
Promise<InteractiveSegmenter>
InteractiveSegmenter.createFromOptions()
Initializes the Wasm runtime and creates a new interactive segmenter from the provided options.
Signature:
static createFromOptions(wasmFileset: WasmFileset, interactiveSegmenterOptions: InteractiveSegmenterOptions): Promise<InteractiveSegmenter>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| wasmFileset | WasmFileset | A configuration object that provides the location of the Wasm binary and its loader. |
| interactiveSegmenterOptions | InteractiveSegmenterOptions | The options for the Interactive Segmenter. Note that either a path to the model asset or a model buffer needs to be provided (via baseOptions). A new InteractiveSegmenter. |
Returns:
Promise<InteractiveSegmenter>
InteractiveSegmenter.segment()
Performs interactive segmentation on the provided single image and invokes the callback with the response. The method returns synchronously once the callback returns. The roi parameter is used to represent a user's region of interest for segmentation.
Signature:
segment(image: ImageSource, roi: RegionOfInterest, callback: InteractiveSegmenterCallback): void;
Parameters
| Parameter | Type | Description |
|---|---|---|
| image | ImageSource | An image to process. |
| roi | RegionOfInterest | The region of interest for segmentation. |
| callback | InteractiveSegmenterCallback | The callback that is invoked with the segmented masks. The lifetime of the returned data is only guaranteed for the duration of the callback. |
Returns:
void
InteractiveSegmenter.segment()
Performs interactive segmentation on the provided single image and invokes the callback with the response. The method returns synchronously once the callback returns. The roi parameter is used to represent a user's region of interest for segmentation.
The 'imageProcessingOptions' parameter can be used to specify the rotation to apply to the image before performing segmentation, by setting its 'rotationDegrees' field. Note that specifying a region-of-interest using the 'regionOfInterest' field is NOT supported and will result in an error.
Signature:
segment(image: ImageSource, roi: RegionOfInterest, imageProcessingOptions: ImageProcessingOptions, callback: InteractiveSegmenterCallback): void;
Parameters
| Parameter | Type | Description |
|---|---|---|
| image | ImageSource | An image to process. |
| roi | RegionOfInterest | The region of interest for segmentation. |
| imageProcessingOptions | ImageProcessingOptions | the ImageProcessingOptions specifying how to process the input image before running inference. |
| callback | InteractiveSegmenterCallback | The callback that is invoked with the segmented masks. The lifetime of the returned data is only guaranteed for the duration of the callback. |
Returns:
void
InteractiveSegmenter.segment()
Performs interactive segmentation on the provided video frame and returns the segmentation result. This method creates a copy of the resulting masks and should not be used in high-throughput applications. The roi parameter is used to represent a user's region of interest for segmentation.
Signature:
segment(image: ImageSource, roi: RegionOfInterest): InteractiveSegmenterResult;
Parameters
| Parameter | Type | Description |
|---|---|---|
| image | ImageSource | An image to process. |
| roi | RegionOfInterest | The region of interest for segmentation. The segmentation result. The data is copied to avoid lifetime limits. |
Returns:
InteractiveSegmenter.segment()
Performs interactive segmentation on the provided video frame and returns the segmentation result. This method creates a copy of the resulting masks and should not be used in high-throughput applications. The roi parameter is used to represent a user's region of interest for segmentation.
The 'imageProcessingOptions' parameter can be used to specify the rotation to apply to the image before performing segmentation, by setting its 'rotationDegrees' field. Note that specifying a region-of-interest using the 'regionOfInterest' field is NOT supported and will result in an error.
Signature:
segment(image: ImageSource, roi: RegionOfInterest, imageProcessingOptions: ImageProcessingOptions): InteractiveSegmenterResult;
Parameters
| Parameter | Type | Description |
|---|---|---|
| image | ImageSource | An image to process. |
| roi | RegionOfInterest | The region of interest for segmentation. |
| imageProcessingOptions | ImageProcessingOptions | the ImageProcessingOptions specifying how to process the input image before running inference. The segmentation result. The data is copied to avoid lifetime limits. |
Returns:
InteractiveSegmenter.setOptions()
Sets new options for the interactive segmenter.
Calling setOptions() with a subset of options only affects those options. You can reset an option back to its default value by explicitly setting it to undefined.
Signature:
setOptions(options: InteractiveSegmenterOptions): Promise<void>;
Parameters
| Parameter | Type | Description |
|---|---|---|
| options | InteractiveSegmenterOptions | The options for the interactive segmenter. A Promise that resolves when the settings have been applied. |
Returns:
Promise<void>