FAQ

How to convert ImageFrames and GpuBuffers

The Calculators ImageFrameToGpuBufferCalculator and GpuBufferToImageFrameCalculator convert back and forth between packets of type ImageFrame and GpuBuffer. ImageFrame refers to image data in CPU memory in any of a number of bitmap image formats. GpuBuffer refers to image data in GPU memory. You can find more detail in the Framework Concepts section GpuBuffer to ImageFrame Converters. You can see an example in:

• object_detection_mobile_cpu.pbtxt

How to visualize perception results

The AnnotationOverlayCalculator allows perception results, such as bounding boxes, arrows, and ovals, to be superimposed on the video frames aligned with the recognized objects. The results can be displayed in a diagnostic window when running on a workstation, or in a texture frame when running on device. You can see an example use of AnnotationOverlayCalculator in:

• face_detection_mobile_gpu.pbtxt.

How to run calculators in parallel

Within a calculator graph, MediaPipe routinely runs separate calculator nodes in parallel. MediaPipe maintains a pool of threads, and runs each calculator as soon as a thread is available and all of its inputs are ready. Each calculator instance is only run for one set of inputs at a time, so most calculators need only to be thread-compatible and not thread-safe.

In order to enable one calculator to process multiple inputs in parallel, there are two possible approaches:

1. Define multiple calculator nodes and dispatch input packets to all nodes.
2. Make the calculator thread-safe and configure its max_in_flight setting.

The first approach can be followed using the calculators designed to distribute packets across other calculators, such as RoundRobinDemuxCalculator. A single RoundRobinDemuxCalculator can distribute successive packets across several identically configured ScaleImageCalculator nodes.

The second approach allows up to max_in_flight invocations of the CalculatorBase::Process method on the same calculator node. The output packets from CalculatorBase::Process are automatically ordered by timestamp before they are passed along to downstream calculators.

With either aproach, you must be aware that the calculator running in parallel cannot maintain internal state in the same way as a normal sequential calculator.

Output timestamps when using ImmediateInputStreamHandler

The ImmediateInputStreamHandler delivers each packet as soon as it arrives at an input stream. As a result, it can deliver a packet with a higher timestamp from one input stream before delivering a packet with a lower timestamp from a different input stream. If these input timestamps are both used for packets sent to one output stream, that output stream will complain that the timestamps are not monotonically increasing. In order to remedy this, the calculator must take care to output a packet only after processing is complete for its timestamp. This could be accomplished by waiting until input packets have been received from all inputstreams for that timestamp, or by ignoring a packet that arrives with a timestamp that has already been processed.

How to change settings at runtime

There are two main approaches to changing the settings of a calculator graph while the application is running:

1. Restart the calculator graph with modified CalculatorGraphConfig.
2. Send new calculator options through packets on graph input-streams.

The first approach has the advantage of leveraging CalculatorGraphConfig processing tools such as "subgraphs". The second approach has the advantage of allowing active calculators and packets to remain in-flight while settings change. Mediapipe contributors are currently investigating alternative approaches to achieve both of these advantages.

How to process realtime input streams

The mediapipe framework can be used to process data streams either online or offline. For offline processing, packets are pushed into the graph as soon as calculators are ready to process those packets. For online processing, one packet for each frame is pushed into the graph as that frame is recorded.

The MediaPipe framework requires only that successive packets be assigned monotonically increasing timestamps. By convention, realtime calculators and graphs use the recording time or the presentation time as the timestamp for each packet, with each timestamp representing microseconds since Jan/1/1970:00:00:00. This allows packets from various sources to be processed in a globally consistent order.

Normally for offline processing, every input packet is processed and processing continues as long as necessary. For online processing, it is often necessary to drop input packets in order to keep pace with the arrival of input data frames. When inputs arrive too frequently, the recommended technique for dropping packets is to use the MediaPipe calculators designed specifically for this purpose such as FlowLimiterCalculator and PacketClonerCalculator.

For online processing, it is also necessary to promptly determine when processing can proceed. MediaPipe supports this by propagating timestamp bounds between calculators. Timestamp bounds indicate timestamp intervals that will contain no input packets, and they allow calculators to begin processing for those timestamps immediately. Calculators designed for realtime processing should carefully calculate timestamp bounds in order to begin processing as promptly as possible. For example, the MakePairCalculator uses the SetOffset API to propagate timestamp bounds from input streams to output streams.

Can I run MediaPipe on MS Windows?

Currently MediaPipe portability supports Debian Linux, Ubuntu Linux, MacOS, Android, and iOS. The core of MediaPipe framework is a C++ library conforming to the C++11 standard, so it is relatively easy to port to additional platforms.