Vector Databases

FAISS for Multi-Modal Retrieval: Implementing Hybrid Image-Text Search with Custom Distance Metrics

Nov 7, 2026 · 2 views
pythonvector databaseNLPComputer Visionhybrid searchFAISSMulti-Modal Search

Building search systems that understand both what an image looks like and what its text says is a significant leap forward in user experience. For intermediate to advanced developers, the challenge often lies in combining these disparate data sources into a single, coherent retrieval pipeline. While specialized multi-modal embedding models exist, there is immense value in understanding how to implement hybrid search using low-level libraries like FAISS (Facebook AI Similarity Search). This post explores how to construct a system that retrieves results based on a weighted combination of visual and textual similarity, complete with custom distance metrics to fine-tune relevance.

Why Hybrid Search?

Pure text search misses visual context, while pure image search (using embeddings from models like CLIP) can miss semantic nuances found in metadata or captions. Hybrid search addresses this by computing separate similarity scores for the text and image components of a query and document, then fusing them. In FAISS, this requires managing two separate indexes—one for text embeddings and one for image embeddings—and implementing a logic layer that combines their results.

Setting Up the Multi-Modal Pipeline

To begin, we assume you have pre-computed vectors for your dataset. For this example, we use 512-dimensional embeddings, a common output size for models like CLIP ViT-B/32. We will create two distinct FAISS indices:

  1. Text Index: Using IndexFlatIP (Inner Product) for cosine similarity.
  2. Image Index: Also using IndexFlatIP for simplicity, though IndexIVFFlat is recommended for larger datasets.

import faiss
import numpy as np

# Define embedding dimensions
dim = 512

# 1. Create Text Index (Inner Product)
index_text = faiss.IndexFlatIP(dim)

# 2. Create Image Index (Inner Product)
index_image = faiss.IndexFlatIP(dim)

# Example Data Generation
num_docs = 1000
np.random.seed(42)
text_embeddings = np.random.rand(num_docs, dim).astype("float32")
image_embeddings = np.random.rand(num_docs, dim).astype("float32")

# Normalize vectors for Cosine Similarity (required for IP to act as Cosine)
faiss.normalize_L2(text_embeddings)
faiss.normalize_L2(image_embeddings)

# Add vectors to indices
index_text.add(text_embeddings)
index_image.add(image_embeddings)

Implementing Custom Distance Metrics and Fusion

The core of hybrid search is the fusion strategy. A common approach is Linear Combination, where the final score is a weighted sum of the individual scores. However, raw scores from different indices may not be on the same scale. We need a Custom Distance Metric function that normalizes these scores before combination.

Below is a Python function that performs the hybrid search. It accepts weights alpha (for text) and beta (for image), where alpha + beta = 1.0.


def hybrid_search(index_t, index_i, query_text_vec, query_img_vec, k=10, alpha=0.5):
    """
    Performs hybrid search by combining text and image similarity scores.
    
    :param index_t: FAISS Index for text embeddings
    :param index_i: FAISS Index for image embeddings
    :param query_text_vec: Normalized text query vector (1, dim)
    :param query_img_vec: Normalized image query vector (1, dim)
    :param k: Number of results to return
    :param alpha: Weight for text similarity (1-alpha is weight for image)
    :return: List of (index, combined_score) tuples
    """
    # Normalize query vectors if not already
    faiss.normalize_L2(query_text_vec)
    faiss.normalize_L2(query_img_vec)

    # Perform search on both indices
    # Note: FAISS returns indices and distances (scores)
    D_text, I_text = index_t.search(query_text_vec, k)
    D_image, I_image = index_i.search(query_img_vec, k)

    # We need to fuse results. Since I_text and I_image might differ,
    # we retrieve the top-k from both and merge them.
    candidates = {}

    # Collect text scores
    for idx, score in zip(I_text[0], D_text[0]):
        if idx != -1:
            candidates[idx] = {'text_score': float(score), 'image_score': 0.0}

    # Collect image scores
    for idx, score in zip(I_image[0], D_image[0]):
        if idx != -1:
            if idx in candidates:
                candidates[idx]['image_score'] = float(score)
            else:
                candidates[idx] = {'text_score': 0.0, 'image_score': float(score)}

    # Apply Custom Weighted Sum (Linear Combination)
    # Final Score = alpha * TextScore + (1 - alpha) * ImageScore
    weighted_results = []
    for doc_id, scores in candidates.items():
        combined_score = (alpha * scores['text_score']) + ((1 - alpha) * scores['image_score'])
        weighted_results.append((doc_id, combined_score))

    # Sort by combined score descending (FAISS returns higher scores as better for IP)
    weighted_results.sort(key=lambda x: x[1], reverse=True)
    
    return weighted_results[:k]

# Example Usage
query_text = np.random.rand(1, dim).astype("float32")
query_img = np.random.rand(1, dim).astype("float32")

results = hybrid_search(index_text, index_image, query_text, query_img, k=5, alpha=0.7)
for doc_id, score in results:
    print(f"Doc ID: {doc_id}, Combined Score: {score:.4f}")

Optimization and Advanced Strategies

The linear combination above is simple but can be improved. For production systems, consider:

  • Reciprocal Rank Fusion (RRF): Instead of summing raw scores, sum the inverse ranks. This is robust to scale differences between modalities.
  • Index Selection: For datasets exceeding 100k vectors, replace IndexFlatIP with IndexIVFPQ (Inverted File with Product Quantization) to reduce memory footprint and increase search speed. You will need to retrain these indices on your specific data distribution.
  • Dynamic Weighting: Use a classifier to determine if a query is more text-heavy or image-heavy and adjust alpha dynamically.

Conclusion

Implementing multi-modal retrieval with FAISS offers granular control over how your search engine interprets user intent. By decoupling the text and image indices and applying custom fusion logic, you can create a search experience that is both semantically rich and visually intuitive. Start with linear weighting, measure your hit rates, and iterate toward more complex fusion strategies like RRF or learning-to-rank models. With FAISS’s speed, even real-time hybrid search on large-scale datasets is achievable.

Share:
Related Posts