Developer/MachineLearning/Embeddings/Fundamentals/choosing-an-embedding-model-and-keeping-it-consistent

Choosing an Embedding Model, and Keeping It Consistent Between Indexing and Querying

Concept

An embedding model turns unstructured content (text, and increasingly images/audio) into a dense vector — a fixed-length list of floats positioned in a high-dimensional space so that semantically similar inputs land close together under some distance metric (cosine similarity or dot product, most commonly). This is the substrate underneath both AI retrieval (RAG's document search, see chunking strategy and retrieval quality) and classical ML similarity/recommendation use cases: same underlying representation, different consumer.

The model choice matters more than almost any other component in a retrieval system, because retrieval quality is bounded by how well the embedding space actually separates "relevant" from "irrelevant" for the domain in question — a downstream reranker or a bigger vector index can't fix a poorly-separated embedding space. A detail that trips up otherwise-correct implementations: several providers (Cohere notably) distinguish between embedding a document being indexed and embedding a query being searched, via an explicit input_type parameter (search_document vs. search_query) — the model produces a different vector for the same text depending on which role it's told the text is playing, because a good query embedding and a good document embedding aren't optimizing for the same objective. OpenAI's embedding API, by contrast, makes no such distinction — the same call embeds text the same way regardless of intended use.

Tradeoffs

Model / provider	Benefit	Cost
OpenAI (`text-embedding-3-large`/`-small`)	Simple API (no query/document distinction to get wrong), strong general-purpose quality, large ecosystem support	No dedicated query-vs-document mode — retrieval-specific optimization has to happen elsewhere (prompt-shaping the query text itself)
Cohere (`embed-v3`/later)	Explicit `input_type` for document vs. query embeddings, tuned specifically for retrieval	An extra parameter that's easy to get backwards (embedding a query with `search_document` mode, or vice versa) — a silent quality regression, not an error
Open-source, self-hosted (e.g. e5, BGE families)	No per-call cost, full control over versioning and deployment, can be fine-tuned on domain data	Operational burden (serving infra, GPU/CPU cost, model updates); generally behind top proprietary models on out-of-the-box benchmark quality without fine-tuning
Switching embedding models on an existing index	Access to a better/cheaper newer model	Every existing vector must be re-embedded and the index rebuilt — different models produce vectors in unrelated spaces (different dimensionality, different geometry), so old and new vectors are not comparable and mixing them silently returns nonsense similarity scores

Embedding-similarity research also shows meaningful divergence between providers' output for the same input: OpenAI's models produce embeddings more similar to Mistral's, while Cohere's embed-english-v3.0 is closer to e5-large-v2 — the point being that "generic sentence-similarity performance" and "retrieval-optimized performance" aren't interchangeable claims, and a model that scores well on one benchmark family may not transfer cleanly to another use case.

When to use / when not to

Use embeddings any time content needs to be compared, clustered, or searched by meaning rather than exact keyword match — RAG document retrieval, semantic search, deduplication, recommendation-by-similarity.
Pick a retrieval-optimized model (with an explicit query/document mode, like Cohere's) when the dominant use case genuinely is asymmetric search — short queries against long documents — since that's the exact case the mode distinction is designed for.
A general-purpose model (OpenAI-style, no mode distinction) is fine, and simpler to operate correctly, when the use case is symmetric similarity (comparing two documents to each other, deduplication, clustering) rather than short-query-against-long-document search.
Don't switch embedding models on a live index without a rebuild plan — a partially-migrated index (some vectors from the old model, some from the new) produces similarity scores that mix two unrelated coordinate systems and silently degrades retrieval quality without throwing any error.
Don't assume higher dimensionality (Cohere's 1024 vs. OpenAI's 3072, per current model specs) predicts better retrieval quality on its own — dimensionality trades off storage/compute cost against representational capacity, but empirical retrieval benchmarks on the actual domain matter more than the raw number.

Common pitfall

Using the wrong input_type (or skipping the query/document distinction entirely on a model that supports it) — embedding a user's search query with the same mode used for indexing documents. This doesn't error; it silently produces a worse-than-expected embedding, because the model optimized document-mode vectors and query-mode vectors for different objectives, and using the wrong one for the wrong side of the search leaves relevance quality on the table without any visible failure signal. The second version of this pitfall is at migration time: re-embedding only newly-added documents with a new model while leaving the existing index on the old model's vectors, which silently corrupts similarity search for the whole index rather than just degrading it for the new documents.

Engineering Lens

The embedding model is infrastructure that every downstream retrieval decision inherits — a bigger index, a better reranker, or more careful chunking all operate on top of whatever separation the embedding space already provides, and none of them can recover relevance that the embedding step never captured. The two operational disciplines that actually protect a retrieval system in production are keeping indexing and querying on the exact same model version (including the query/document mode, where the provider distinguishes it), and treating a model swap as a full-index migration event, never an incremental one. Getting those two things right is a bigger lever on retrieval quality than most reranking or prompt-engineering work layered on top.