Jayu

What it does

Jayu is a revolutionary personal assistant that seamlessly integrates Gemini's capabilities with on-screen interaction. Breaking the limits of what an LLM should be able to do, Jayu utilizes Gemini to provide a user-centered experience, for everyone from tech-savvy users to the technologically challenged. No docker container, no complex interface, and no other LLM or VLM besides Gemini. Speech-to-text, text-to-speech, and gesture recognition capabilities are built in for usability.

Jayu’s strength lies in its unique ability to answer prompts with your screen as context and interact with on-screen elements. From writing code based on a diagram to directly interacting with apps to reading out live translations, Jayu can do it all.

A Flash model is used as the command center. After receiving instructions from the user, the model uses function calling to call other Gemini models to assist with its task if necessary. Through prompt engineering, Flash models interact directly with Chrome and answer quick questions, while Pro models are trained to use Gemini’s powerful vision capabilities to analyze app windows. And Gemini’s object detection capabilities allow Jayu to click buttons it sees on the screen.

We realize the security risks of having access to your screen or files; Jayu cannot access folders or any apps that are not shown to it. Jayu will only look at your screen if directly prompted to do so. Jayu also does not retain any memory or logs of images or recordings.