A persistent attribute of AI-powered applications is their propensity to generate text: lots and lots of text. Text requires scanning and scrolling for the useful bits and too much of it gets pretty monotonous pretty quick. So I've been working on more visual replies for my personal AI, Ask LukeW and just launched a big improvement.
Ask LukeW provides answers to digital product design and strategy questions using my corpus of thousands of articles, hundreds of presentations, and (more recently) thousands of images. To make the images I've created for my articles and talks searchable, an ingestion pipeline watches for new image uploads. When an image is added to my site, an AI model examines its contents and produces a title and description for the image. The title and description are both saved, and each is turned into an embedding so the image is also searchable semantically.
When someone asks a question, a retrieval system not only searches for any relevant images semantically (using the embeddings) but also using more traditional keyword search. This brings back a ranked list of images that the AI answering someone's question can choose to include in its response. The better the retrieval, the more likely an answer can include relevant images that break up what would otherwise be a wall of text.
For instance here's a comparison of the same question without a relevant image and with one. Most prefer the answer plus visuals version.
But with this system, images were rarely included in replies despite there being plenty of good candidates. Why? Looking at the titles and descriptions generated by AI during the aforementioned ingestion process provides some answers. Consider this image from an article about off-canvas responsive images.
When ingested this image was given a title of "Green block layout comparison". I mean, that's technically correct but who is going to ask a question about green block layouts? So despite a reasonable title, this image would pretty much never show up in results. Thankfully, we can learn how it should show up by looking at the article in which the image appeared.
For the past 30 years, when I've added images to my articles, I always included an ALT tag: a very common accessibility best practice that gives screen readers and more a useful description of images in Web pages. The ALT tag for this particular graphic was "Why Off Canvas Layouts?". Same image, totally different description. While neither is perfect, both descriptions are useful for retrieval.
As I often say "AI begets more AI" so the answer (of course) was to use a fast, yet smart, AI model to combine any existing ALT tags for images with their previously generated titles. For the image above that became: "Green block layout comparison showing why off canvas layouts are used". Wordy, but much better.
And since we can be wordy, the model writing the new title for each image can now also make use of the full visual description if it wants to. Here's another image to illustrate that.
The ingestion pipeline titled this image "Image Prompt Enhance Feature". The ALT tag was "Reve enhance feature". But the new title became "Reve enhance feature showing a prompt editor expanding a brief Spider-Man prompt into a detailed version" by pulling a bit from the full description. Much better.
So what's the impact of all this? More answers with images of course. Sticking with our example above, here's how images now show up in What are off canvas layouts?
Of course, this system needs to be dynamic. If I upload an image, it gets titled from the picture alone. If that same image shows up in a later article, it gets retitled. Updating ALT tags in old posts does the same thing. Lastly, If I ever rename an image manually, the pipeline won't overwrite it. AI begets more AI, but it should still defer to us humans for the last word (for now).