The Power and Pitfalls of Vector-Based Image Search

Vector-based image search is implemented by first converting images into numerical vector embeddings using models like clip-ViT-B-32. A vector database such as Milvus is used to store these embeddings in a collection, which requires defining a schema and creating an index for efficient querying. Data is inserted into the collection, often in batches, to build the searchable database of image vectors. An Approximate Nearest Neighbor (ANN) search is then performed by providing a query image's vector to find the most visually similar items based on a metric like cosine similarity. While this method is effective at finding visually identical images, it can also return contextually unrelated items that share visual characteristics, demonstrating a key limitation.