Published June 16, 2026
When I told a friend the third sense of my memory system would read the text of whatever window I had in focus, the first question was the obvious one: “So it does OCR? Or vision?” It does neither, and the reason I didn’t take that path is the design decision I find most interesting in the whole project.
The naïve approach would be to screenshot the foreground window every few seconds, run it through a local vision-language model, get a description and extract themes. The technology works, and there are commercial products doing exactly this today. Three problems made me reject it.
One: it costs resources I don’t have. Running a vision model locally on a normal developer machine means choosing between it and the language model that’s already doing the curation, because they don’t fit together. Cloud is out by hard rule: nothing in this system phones home.
Two: it produces noisy, opinionated output. A vision model looking at a screenshot doesn’t say “the user is reading about React Hooks.” It says “a webpage with a header bar, a left navigation menu, dark-theme code blocks, a comment section below.” Most of what it generates describes the interface, not the topic. That’s a poor signal-to-noise ratio for a system trying to extract one theme.
Three: it’s the wrong abstraction. The text on your screen isn’t a picture, it’s text. Treating it as a picture means destroying the structure at the bitmap and then trying to recover it with statistics.
So I went one layer down. Windows has an accessibility API, which it calls UI Automation, and macOS and Linux have something similar. It exists so screen readers for blind users can announce what’s on screen. Browsers, document editors, native apps and even most Electron apps expose a tree of text through it. The same API that lets a screen reader say “Heading: React Hooks. Paragraph: useState lets you add state to a function component” is the one my screen sense queries.
That gives me clean, structured text, attribution to the app that produced it, and a fast read with no GPU involved. What I lose is anything in a canvas, an image, a video or a game, since those apps expose no accessibility text. I’m fine with that. If I want the memory to learn from those, I should do it on purpose, by pasting a screenshot or writing a note, rather than let a vision model guess.
There’s a privacy side to this too. The accessibility tree skips password fields by default: they’re flagged at the API level, and a reader that respects the flag never sees the contents. A vision model pointed at a banking app would see the rendered balance, the account number and everything else on screen. The accessibility tree doesn’t hand that over.
So there is no OCR and no vision, only the protocol that has let screen readers describe interfaces for thirty years, repurposed to feed a personal memory. The memory knows what I’m reading without seeing my screen. The open question is how much it misses in apps that expose little or no accessibility text, and that gap is what I’ll be watching.
Javier
The screen sense, which reads text and never takes screenshots, is part of SUPER DARA.