When Labels Say No, But Embeddings Say Yes

Traditional evaluation metrics for search systems can be flawed, as they often penalize results that are semantically similar but have different labels. For example, a system might be marked as incorrect for finding a visually similar Lamborghini when a user searched for a Ferrari, simply because the brand doesn't match. An experiment using the CLIP model on architectural images investigates whether the model's "mistakes" are actually meaningful connections between visually similar but differently labeled building styles. By comparing standard label-based precision with a new metric based on shared architectural features, the research questions whether we should measure retrieval quality by degrees of similarity rather than a simple right-or-wrong label match.