Toebeans

What it does

Toebeans is a fun app for getting different types of fortunes for your pet cat. It can generate paw readings, horoscopes, tell you about their past lives, and you can ask it some predefined questions. The app is currently a work-in-progress, and is planned to be released in the fall. Toebeans is a Compose Multiplatform app that targets both Android and iOS. On the Android side it uses MediaPipe's object detection to detect the cats paws either by consuming the camera feed or from a selected image.

Once we have a picture of the paw (toebeans) the user can then get different fortunes for their cat. On Android it uses VertexAI and Gemini to generate the response from the given fortune prompt. On iOS it uses Firebase Functions to call the Gemini API. This is done because currently the VertexAI package for iOS is written in swift and not compatible with Kotlin Multiplatform. Toebeans also uses Firebase Remote Config to be able to adjust the prompt parameters and model settings remotely. As well as Firebase Auth for user authentication (currently disabled).

The user is also able to "modify" their original fortune response to a different type. The app then takes the original response and tells Gemini to rewrite it in a different style.

The plan for the future is to use GenKit instead of VertexAI and Firebase functions. We are also in the process of tuning a Gemini model by using the AI studio.