TwilightNav

What it does

The app utilizes the Gemini API to enhance user interaction through two key functions:

Content Analysis and Structuring: The Gemini API analyzes the HTML format of a webpage, converting it into a hierarchical tree structure. In this structure, each node represents a specific content element, containing both a description of that element and a summary of its child elements. This organization allows for efficient processing and understanding of webpage content.

Intent Recognition and Navigation: The Gemini API also plays a crucial role in understanding user instructions. It processes voice input to detect the user's intent and identifies the corresponding target node within the tree structure. The API categorizes user commands into six intents: navigating to a website, summarization, reading content, querying information, clicking elements, and filling out forms.

By leveraging these capabilities, the app enables users to perform a variety of web-based tasks through voice commands and gestures, making the browsing experience more accessible and intuitive.