How Vision Language Models Are Trained from “Scratch”

Modern Vision Language Models (VLMs) are created by fine-tuning pre-trained text-only models, rather than training them from scratch. This process involves a standard architecture with a frozen image backbone like a Vision Transformer (ViT), an adapter layer, and the language model. The adapter layer, such as a Q-Former, uses learnable queries and cross-attention to translate image embeddings into a format the language model can understand. This alignment is achieved by training the adapter on image-text pairs using specific loss functions like Image-Text Contrastive loss.