What Is a VLM? How Computer Vision and LLMs Combine

A VLM (Vision Language Model) is an AI model built to understand images and text together in the same representation space — show it a photo, and it can describe what’s in it or answer questions about it. Unlike deep learning, where image processing and language processing have historically evolved separately, a VLM’s distinguishing feature is connecting both domains within a single model.

The basic architecture of a VLM

A typical VLM has three parts.

1. Vision encoder — converts an image into numerical vectors (visual tokens). The image encoder from a model like CLIP is often reused here.
2. Connector (projection layer) — transforms (aligns) the visual tokens produced by the vision encoder into an embedding space the language model can understand. Usually implemented as a relatively simple linear layer (MLP).
3. LLM (large language model) — takes the transformed visual tokens together with text tokens as input and generates text, such as a description of or answer about the image.

In short, a VLM works by building a “bridge” that turns an image into another form of token the language model can read.

CLIP: training that places images and text side by side

CLIP (Contrastive Language-Image Pretraining) is a contrastive-learning model trained on hundreds of millions of image-caption pairs so that an image and text describing the same content end up close together in vector space. CLIP’s resulting image encoder became a standard component reused as the vision encoder across many later VLMs.

LLaVA: a leading example of connecting a vision encoder to an LLM

LLaVA connects a frozen CLIP vision encoder and a frozen LLM through a small, trainable MLP connector. Training happens in two stages.

1. Stage 1 — train only the connector: the vision encoder and LLM stay fixed while only the connector, which “translates” image tokens into something the language model can understand, is trained first.
2. Stage 2 — fine-tune the connector and LLM together: the connector and LLM are fine-tuned together on instruction data, giving the model the ability to answer questions about images naturally.

This approach is far cheaper computationally than training the image encoder and LLM together from scratch.

Where VLMs are used

• Generating image descriptions (captioning), visual question answering (answering questions about a photo)
• Reading and summarizing documents, tables, and chart images
• In Physical AI, the vision-language link a robot needs to understand a scene from its camera and decide its next action

Limitations and caveats

• VLMs aren’t immune to “hallucination” — plausibly making up content that isn’t actually in the image.
• A vision encoder may not handle image types absent from its training data well (e.g. specialized sensor imagery).
• Image resolution, the number of images that can be processed, and response accuracy vary a lot by model, so check benchmarks and official documentation before actual deployment.

Conclusion

A VLM works by connecting an image encoder and a language model through a connector, turning visual information into tokens the language model can understand. If CLIP provided the standard vision encoder that aligns images and text, models like LLaVA are the leading example of efficiently connecting that encoder to an LLM.