Why We Fine-Tuned SigLip (And Why That’s Not Always the Right Call)

A team at Alma Media developed an image classification system to automatically tag real estate photos with room types. They provide a framework for deciding between using a third-party VLM API and training a custom model, weighing factors like cost, data requirements, and the need for reliable confidence scores. The authors explain their choice to fine-tune the SigLip model using LoRA to solve an under-labeling problem they faced with a frozen model. The discussion also covers how to select a foundation model like SigLIP or DINO based on business goals and the model's pre-training objectives.