When the model reads images

The model doesn't really see

When you send a photo to GPT-4o or Claude and it tells you what it represents, there's no "eye" in the model. No visual system, no object detector. The model doesn't "see" an image — it reads a sequence of vectors.

That's the entire trick of modern multimodality: transform any type of data (image, audio, video) into a representation that resembles text tokens. After this transformation, the standard Transformer does the rest.

A note on this chapter's scope. We're talking here about models that understand images (describe them, answer questions, read a chart). For models that generate images from text — Stable Diffusion, DALL-E, Midjourney — the architecture is different and is the subject of chapter 21.

ViT: cut the image into patches

The reference architecture for images is called ViT (Vision Transformer), proposed by Google in 2020.

The idea is disconcertingly simple: cut the image into small squares (patches) of fixed size — 16×16 pixels by default. Each patch is flattened into a vector, then projected into the Transformer's embedding space.

A 224×224 pixel image, cut into 16×16 patches, produces 196 patches — 196 tokens. These tokens are sent to the Transformer exactly like text tokens. Attention processes them, relates them, extracts relevant features.

The position of each patch in the image is encoded via a positional embedding, exactly like for text tokens.

Explore the patches

Here's a 16×16 image (a simplification — ViT-Base works on 224×224). Change the patch size to observe how resolution and token count vary.

The image is split into square patches; each patch becomes a token via a linear projection. The Transformer no longer cares whether the input is text or an image grid — it processes the sequence the same way.

The trade-off shows up as soon as you move the slider: smaller patches give a better-defined image, but the token count explodes — and since attention costs O(n²) (chapter 04), halving the patch size quadruples the number of tokens and multiplies the cost by sixteen. That's exactly the trade every vision model makes: enough patches to read a chart, not so many that inference becomes ruinous.

The special [CLS] token

In early ViTs, a special [CLS] token is added at the beginning of the sequence. After attention has circulated information among all patches, the [CLS] vector is used as a global representation of the image.

It's the one sent to a classification head to answer "what is this?"

Today's multimodal models — the GPT-4V, Claude 3 and Gemini generation opened the way in 2023-2024, and every major model can do it now — proceed differently: patch tokens are concatenated with text tokens in one and the same sequence. From there, nothing special is needed: ordinary self-attention lets every word look at every piece of image, and vice versa. That's what makes the approach so economical — no dedicated machinery, you reuse the Transformer as is.

A variant exists where the image stays in its own separate sequence and the text queries it through cross-attention (that's the choice made by Flamingo, or Llama 3.2 Vision). More modular, but less simple.

CLIP: aligning images and text

Before you can mix images and text in the same Transformer, image embeddings and text embeddings must live in the same vector space.

That's the problem CLIP (OpenAI, 2021) solved. CLIP trains two encoders in parallel — one for images, one for text — with a single objective: bring the representations of an image and its caption closer together.

After training on hundreds of millions of (image, text) pairs, CLIP produces spaces where "a photo of a cat" and an actual photo of a cat have close vectors. This property is what lets a LLM "understand" an image injected into its context.

The architecture of a vision-language model

Current multimodal LLMs are generally composed of:

1. A visual encoder (often a pre-trained ViT) that produces image tokens.
2. A projector (a small MLP) that maps image tokens into the LLM's embedding space.
3. The LLM which receives the mixed sequence (image tokens + text tokens) and generates the response.

This "projector" is often the only part trained when adapting a text LLM to multimodal — the rest stays frozen.

What the model actually sees

It's tempting to imagine the model has deep "visual understanding." In reality, here's what happens:

Each 16×16 patch is first flattened: 16 × 16 × 3 colour channels = 768 values, lined up in a single list. That list then goes through a learned linear projection — a matrix, trained along with the rest of the model, that turns it into an embedding vector (768 dimensions for ViT-Base). So it is not an average of the pixels: nothing is squashed, the patch's internal layout survives in the vector. But the representation is still very local — that vector knows nothing about what's happening in the rest of the image.

It's the attention between all these vectors that reconstructs spatial relationships, detects edges, recognizes shapes. The model has no built-in concept of "circle" or "face" — it discovers them statistically.

That's why visual LLMs can be surprising on tasks simple for humans (counting objects, distinguishing left/right) but remarkable on high-level tasks (interpreting a chart, reading a prescription).

What about audio?

The same principle — converting any modality into a token sequence — applies to voice. OpenAI's Whisper (2022) remains the open-source reference for speech-to-text transcription — natively multimodal models now do better, but they stay closed. Its architecture is an encoder-decoder Transformer, exactly like a translation model.

The trick: convert the audio signal into a Mel spectrogram — a 2D image where the vertical axis is frequency and horizontal is time. Each little square of this spectrogram becomes an input token, just like ViT patches for images. Whisper then produces text tokens as output.

For voice generation (text-to-speech), the principle reverses: generate audio tokens from text. ElevenLabs, OpenAI TTS, Suno (for music) all use Transformers trained to predict the next audio token. A loved one's cloned voice is exactly the tokenization of a few minutes of recording used as conditioning.

The leap came from natively multimodal voice models, pioneered by GPT-4o realtime and Gemini Live in 2024. These models no longer round-trip through text internally: they reason directly in a space that mixes text tokens and audio tokens. Hence low latency (~300 ms) and natural prosody — the model can smile while speaking, because it never left the audio domain.

The distinction matters, because plenty of voice assistants aren't that. Claude's voice mode, for instance, chains three separate pieces: transcription, then a text model, then speech synthesis. It works very well, but the model never perceives your voice — only the text someone made out of it.

As with vision, the underlying architecture is still a Transformer. The only difference is the modality of the token.

Tokens: a universal currency

The true lesson of multimodality is that the token is a universal abstraction.

Text → tokens.
Images → tokens (patches).
Audio → tokens (sliced spectrogram).
Molecules → tokens (atoms).

As soon as you can convert a modality into a sequence of dense vectors, a Transformer can process it. That's why the same architectures that revolutionized NLP are now revolutionizing vision, audio, biology, and physics.

The Transformer is a token engine. Researchers keep inventing new ways to tokenize the world.

Which makes one question increasingly awkward. The same model reads text, describes a photo, transcribes a voice and writes code: on what basis do you decide it's better than another? You need a score — and every score is a reduction. The next chapter looks at what benchmarks actually measure, and above all at what they inevitably miss.

Updated August 2, 2026