Companion 1

What it does

"Companion" is a Chrome extension designed to make the web more accessible and user-friendly, particularly for individuals with visual impairments, cognitive disabilities, or language barriers. The app leverages the Gemini API to simplify complex web content, making it more digestible and easier to understand.

The core idea of Companion is to grant Gemini direct access to your current tab's web content, enabling it to provide relevant answers to the exact content you are currently viewing. Currently, the app supports a basic multi-modality with Gemini, allowing you to prompt Gemini with an image and text and receive a text-based response. Future plans include expanding this capability to process multiple images, videos, and audio, as well as generating images to help illustrate web content.

A feature that exemplifies the direction of the app is its ability to understand and respond to voice commands in hover mode. In this mode, Gemini interprets voice commands in relation to user-selected content, providing context-aware responses that align with the ongoing conversation.

Users can also adjust Gemini's responses according to their preferences. For example, you can ask Gemini to simplify information, offer follow-up questions to deepen understanding, or translate content into chosen languages. This extension aims not only to make the web more navigable but also to enhance daily experiences by removing barriers to digital content.