MPPImageSegmenter
@interface MPPImageSegmenter : NSObject
@brief Class that performs segmentation on images.
The API expects a TFLite model with mandatory TFLite Model Metadata.
-
Get the category label list of the
ImageSegmenter
can recognize. For CATEGORY_MASK type, the index in the category mask corresponds to the category in the label list. For CONFIDENCE_MASK type, the output mask list at index corresponds to the category in the label list. If there is no labelmap provided in the model file, empty array is returned.Declaration
Objective-C
@property (nonatomic, readonly) NSArray<NSString *> *_Nonnull labels;
-
Creates a new instance of
ImageSegmenter
from an absolute path to a TensorFlow Lite model file stored locally on the device and the defaultImageSegmenterOptions
.Declaration
Objective-C
- (nullable instancetype)initWithModelPath:(nonnull NSString *)modelPath error:(NSError *_Nullable *_Nullable)error;
Parameters
modelPath
An absolute path to a TensorFlow Lite model file stored locally on the device.
Return Value
A new instance of
ImageSegmenter
with the given model path.nil
if there is an error in initializing the image segmenter. -
Creates a new instance of
ImageSegmenter
from the givenImageSegmenterOptions
.Declaration
Objective-C
- (nullable instancetype)initWithOptions: (nonnull MPPImageSegmenterOptions *)options error:(NSError *_Nullable *_Nullable)error;
Parameters
options
The options of type
ImageSegmenterOptions
to use for configuring theImageSegmenter
.Return Value
A new instance of
ImageSegmenter
with the given options.nil
if there is an error in initializing the image segmenter. -
Performs segmentation on the provided MPPImage using the whole image as region of interest. Rotation will be applied according to the
orientation
property of the providedMPImage
. Only use this method when theImageSegmenter
is created with running mode,image
.This method supports segmentation of RGBA images. If your
MPImage
has a source type of.pixelBuffer
or.sampleBuffer
, the underlying pixel buffer must usekCVPixelFormatType_32BGRA
as its pixel format.If your
MPImage
has a source type of.image
ensure that the color space is RGB with an Alpha channel.Declaration
Objective-C
- (nullable MPPImageSegmenterResult *) segmentImage:(nonnull MPPImage *)image error:(NSError *_Nullable *_Nullable)error;
Parameters
image
The
MPImage
on which segmentation is to be performed.Return Value
An
ImageSegmenterResult
that contains the segmented masks. -
Performs segmentation on the provided MPPImage using the whole image as region of interest and invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns.
Rotation will be applied according to the
orientation
property of the providedMPImage
. Only use this method when theImageSegmenter
is created with running mode,image
.This method supports segmentation of RGBA images. If your
MPImage
has a source type of.pixelBuffer
or.sampleBuffer
, the underlying pixel buffer must usekCVPixelFormatType_32BGRA
as its pixel format.If your
MPImage
has a source type ofimage
ensure that the color space is RGB with an Alpha channel.Declaration
Objective-C
- (void)segmentImage:(nonnull MPPImage *)image withCompletionHandler: (nonnull void (^)(MPPImageSegmenterResult *_Nullable, NSError *_Nullable))completionHandler;
Parameters
image
The
MPImage
on which segmentation is to be performed.completionHandler
A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional
ImageSegmenterResult
that contains the segmented masks if the segmentation was successful and an optional error populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block. -
Performs segmentation on the provided video frame of type
MPImage
using the whole image as region of interest.Rotation will be applied according to the
orientation
property of the providedMPImage
. Only use this method when theImageSegmenter
is created withvideo
.This method supports segmentation of RGBA images. If your
MPImage
has a source type of.pixelBuffer
or.sampleBuffer
, the underlying pixel buffer must usekCVPixelFormatType_32BGRA
as its pixel format.If your
MPImage
has a source type ofimage
ensure that the color space is RGB with an Alpha channel.Declaration
Objective-C
- (nullable MPPImageSegmenterResult *) segmentVideoFrame:(nonnull MPPImage *)image timestampInMilliseconds:(NSInteger)timestampInMilliseconds error:(NSError *_Nullable *_Nullable)error;
Parameters
image
The
MPImage
on which segmentation is to be performed.timestampInMilliseconds
The video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.
Return Value
An
ImageSegmenterResult
that contains a the segmented masks. -
Performs segmentation on the provided video frame of type
MPImage
using the whole image as region of interest invokes the given completion handler block with the response. The method returns synchronously once the completion handler returns.Rotation will be applied according to the
orientation
property of the providedMPImage
. Only use this method when theImageSegmenter
is created with running mode,video
.This method supports segmentation of RGBA images. If your
MPImage
has a source type of.pixelBuffer
or.sampleBuffer
, the underlying pixel buffer must usekCVPixelFormatType_32BGRA
as its pixel format.If your
MPImage
has a source type ofimage
ensure that the color space is RGB with an Alpha channel.Declaration
Objective-C
- (void)segmentVideoFrame:(nonnull MPPImage *)image timestampInMilliseconds:(NSInteger)timestampInMilliseconds withCompletionHandler: (nonnull void (^)(MPPImageSegmenterResult *_Nullable, NSError *_Nullable))completionHandler;
Parameters
image
The
MPImage
on which segmentation is to be performed.timestampInMilliseconds
The video frame’s timestamp (in milliseconds). The input timestamps must be monotonically increasing.
completionHandler
A block to be invoked with the results of performing segmentation on the image. The block takes two arguments, the optional
ImageSegmenterResult
that contains the segmented masks if the segmentation was successful and an optional error only populated upon failure. The lifetime of the returned masks is only guaranteed for the duration of the block. -
Sends live stream image data of type
MPImage
to perform segmentation using the whole image as region of interest.Rotation will be applied according to the
orientation
property of the providedMPImage
. Only use this method when theImageSegmenter
is created with running mode,liveStream
.The object which needs to be continuously notified of the available results of image segmentation must confirm to
ImageSegmenterLiveStreamDelegate
protocol and implement theimageSegmenter(_:didFinishSegmentationWithResult:timestampInMilliseconds:error:)
delegate method.It’s required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the segmenter. The input timestamps must be monotonically increasing.
This method supports segmentation of RGBA images. If your
MPImage
has a source type of.pixelBuffer
or.sampleBuffer
, the underlying pixel buffer must usekCVPixelFormatType_32BGRA
as its pixel format.If the input
MPImage
has a source type ofimage
ensure that the color space is RGB with an Alpha channel.If this method is used for classifying live camera frames using
AVFoundation
, ensure that you requestAVCaptureVideoDataOutput
to output frames inkCMPixelFormat_32BGRA
using itsvideoSettings
property.