Why this is a semiotic read-out, not an image classifier

An image classifier is trained on labelled images. It learns a fixed map from pixels to a closed label set, and it only knows the labels it was shown.

This read-out is different in kind. It never saw an image in training. It was trained only on text — to separate discourse communities in the residual stream of a frozen gemma-4-31B. What you are watching is that text-only meaning space being asked to interpret a picture, and succeeding, because gemma-4 already fuses image and word into one representational stream and the read-out taps the shared interpretant: the meaning a sign carries, whatever form it arrived in.

So it does not classify the image. It tells you what the image means to a system that learned meaning from words — a transfer across modality, from a head that was never cross-trained. That is the result.

How to read the three panels

• The words it means — the load-bearing evidence. The image is placed in a shared word space and the nearest words are shown. This is precise, and it is the point.
• The caption it retrieves — the same image state, asked to pick a full sentence out of 10,000 COCO captions it has never been trained against. Open vocabulary, zero training: the image's place in meaning space is precise enough to select a whole description, not just a word.
• The discourse it evokes — a flavour, not a category. The head knows only 35 discourse communities from its training corpus, far too few to label the visual world precisely. Read it as the nearest cultural neighbourhood a picture leans toward (cars into the automotive community; deer and mushrooms into gardening; cats and dogs into the cozy-domestic communities), never as a class label.