1PUL

What it does

1PUL (1st Place You Look, pronounced "one pull") is a Flutter web app (prototype) that inventories items moved by people, and has an interface to search the inventory. In an environment with many items and lots of storage options (like a busy household or an office), it can be challenging to 1) track down the last place an item was seen, and 2) have all the items in mind when trying to solve a problem or fulfill a list. 1PUL can also be used to actively inventory a collection of items.
When the Flutter code detects movement from the camera by comparing frames, it hits the Gemini API with a camera frame and a prompt like, "if there's a person holding an object, and the object is clearly visible, describe the object". Gemini's object description, plus the still image url, camera location, and (optionally) transcribed audible comments from the person, are written to a Google Sheet. The still image is uploaded to a GCS bucket. The capturing device running 1PUL will also briefly show that an item is being inventoried.
Inventory Search is a feature that sends a user query (ex: "what camping gear do I have", "where's my socket wrench") to the Gemini API, along with the full inventory descriptions (thanks for the giant context window!), with a prompt like, "match this user query to the most relevant item descriptions". There's some prompting to get structured output, which the dart code maps back to inventory items to present to the user.