Vespa for Multi-Modal Search: Combining Vector, Keyword, and Geospatial Data in a Single Query

Modern search applications have moved far beyond simple keyword matching. Today’s users expect a seamless experience that understands semantic intent, respects specific terminology, and considers physical location. For example, a user might search for "best vegan restaurants near me," where "near me" is geospatial, "vegan" is a keyword constraint, and "best" implies a semantic ranking based on reviews or image embeddings.

Many traditional architectures struggle to handle these three distinct data types simultaneously without complex microservices orchestration. Vespa, an open-source search and serving engine, addresses this challenge natively. By treating vectors, text, and location as first-class citizens, Vespa allows developers to fuse these signals in a single, high-performance query.

Why Multi-Modal Search Matters

Hybrid search is no longer a novelty; it is a requirement. Pure vector search often lacks the precision needed for exact brand names or technical codes. Pure keyword search fails to capture the semantic nuance of natural language queries. Adding geospatial constraints creates another layer of complexity, as location data is often stored separately from content.

Vespa’s architecture eliminates these silos. It indexes all data types into a unified schema, allowing the rank profile to weigh and combine these signals using custom ranking functions. This results in lower latency and higher relevance compared to federating multiple search engines.

Defining the Schema: Structuring Multi-Modal Data

To enable multi-modal search, your schema must define the appropriate field types. Vespa supports string for text, geopoint for location, and tensor for vector embeddings.

schema products {
    document store {
        field name type string {
            index
        }
        field location type geopoint {
            index
        }
        field embedding type tensor<float>[128] {
            index
        }
        field category type string {
            index
        }
    }
}

Note that the tensor field requires an index to be searchable via vector similarity, while geopoint requires an index for spatial queries.

Crafting the Multi-Modal Query

The power of Vespa lies in its ability to execute a single query that targets multiple data types. You can use the select statement to filter by location, filter by keywords, and perform a vector nearest neighbor search simultaneously.

Consider a query for "luxury hotels in Paris with modern decor." You might structure your query like this:

GET /search/?query=(
    userQuery:(luxury hotel) OR 
    embedding:nearestNeighbor(embedding, [0.1, 0.2, ...]) 
) AND location:geoCircle(48.8566, 2.3522, 5km)
&ranking=queryName

In this example:

1. Keyword Match: userQuery:(luxury hotel) ensures traditional text matching.
2. Vector Match: nearestNeighbor finds semantically similar documents based on the embedded query vector.
3. Geospatial Filter: geoCircle restricts results to within 5km of Paris coordinates.

Ranking Strategies for Fusion

Simply retrieving results from all three sources is not enough; you must rank them coherently. Vespa’s Rank Profile allows you to define custom formulas that blend the scores from different fields.

rank-profile MultiModalRank {
    inputs {
        input<float> vectorWeight: 0.5
        input<float> keywordWeight: 0.3
        input<float> locationWeight: 0.2
    }
    first-phase {
        function<float> vector_score() {
            expression: closestMatchDistance(embedding)
        }
        function<float> keyword_score() {
            expression: bm25(name)
        }
        function<float> location_score() {
            expression: 1.0 - geoDistance(location) / 10000.0
        }
        expression: vectorWeight * vector_score() + 
                    keywordWeight * keyword_score() + 
                    locationWeight * location_score()
    }
}

This formula dynamically weights each component. Developers can tune these weights based on A/B testing to optimize for their specific use case, whether they prioritize semantic relevance or geographic proximity.

Conclusion

Vespa provides a robust, scalable solution for multi-modal search, eliminating the complexity of integrating separate vector, text, and geospatial databases. By leveraging its unified indexing and flexible ranking engine, developers can build highly relevant, context-aware search experiences with low latency. As data becomes more diverse, the ability to fuse these modalities in a single query becomes a critical competitive advantage.