GUI Dog

What it does

You want to access a specific file. For a sighted user, you simply click to enter the Documents folder and click again to open the file. Done.

But what if you're visually impaired? You'd use a blind reader, such as macOS's VoiceOver. So you open a file manager window, and VoiceOver describes, in detail, what it is and can do (every time you use it!). You use complex keyboard actions to navigate to the sidebar. Whenever you interact with your device, VoiceOver repeats information about the focused UI element. You then go down the list of folders, one by one, with VoiceOver announcing every folder's name until you finally hear the correct folder. The process is repeated inside the folder until you find your desired file. Wasn't that exasperating?

A quiet two-click task turns into a yakking, tiresome dozen-action operation.

What if you could complete the same task simply by instructing, "In the Documents folder, open my vaccination document"? My app, GUI Dog, does that.

The user gives GUI Dog precise instructions for a task. GUI Dog provides Gemini with the task to achieve and a list of UI elements it can manipulate. Gemini then specifies what actions to take, and GUI Dog uses the macOS accessibility system to execute them. Together, they work towards completing the task.

GUI Dog aims to be the "seeing eye" for the visually impaired, using Large Language Model technologies to help uplift the disadvantaged, enhancing their interactions with the digital world.