How I Design and Tune Vector Search in Production
Published · Last updated · 6 min read
When I first started building semantic search into retrieval-augmented generation systems, it was easy to view vector databases as magic storage: feed text to an embedding model, save the floats, and run a nearest-neighbor query to get the answer.
In a small demo, that works cleanly. But when I set out to build an intelligent internal documentation and runbook search engine for an engineering team—indexing microservice architecture specs, deployment guides, and post-mortem incident logs—semantic search revealed its real mechanics.
You discover that embedding models trade fidelity for cost, vector stores lock schemas permanently on initialization, and mathematical proximity does not automatically translate into factual accuracy.
This post covers the operational decisions and failure modes I ran into while building and tuning vector retrieval for an engineering knowledge base.
How does embedding dimensionality affect semantic fidelity?
The first decision in any retrieval pipeline is selecting the embedding model. It is tempting to default to the smallest, fastest model available. But that decision has a direct quality consequence.
An embedding model translates raw text into an array of floating-point numbers. In our system, OpenAI’s text-embedding-3-small produces 1,536 dimensions, whereas a local model like sentence-transformers/all-MiniLM-L6-v2 produces 384 dimensions.
Lower dimensionality compresses semantic space. Think of it like taking a contoured 3D topographical map and flattening it into a 2D line drawing:
- High-level concepts still separate cleanly.
- Subtle, fine-grained technical distinctions collapse.
When indexing dense technical documentation, a 384-dimensional space often places distinct concepts (like circuit breaker thresholds versus token bucket rate limits) too close together. The model simply lacks the geometric coordinates needed to isolate specialized engineering terminology.
| Embedding Model | Dimensions | Hosting Mode | Relative Latency | Semantic Nuance |
|---|---|---|---|---|
text-embedding-3-small | 1,536 | Managed API | Network-bound | High (isolates domain concepts) |
all-MiniLM-L6-v2 | 384 | Local CPU / GPU | Sub-millisecond | Moderate (merges subtle distinctions) |
Reaching for the cheapest or fastest embedding model is not a free optimization; it is an explicit trade against retrieval precision.
Why does chunking require overlap and batching?
You cannot feed an entire manual or codebase into an embedding model at once due to context constraints. You must split documents into chunks.
In my pipeline, I set chunk sizes to roughly 500 tokens. But chunking text blindly at character boundaries introduces a common failure mode: an important sentence gets chopped in half, destroying the semantic relationship between the subject and the predicate.
To maintain continuity across chunk boundaries, I configured an overlap of 50 tokens to prevent context fragmentation:
# src/chunker.py
def chunk_document(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
tokens = tokenize(text)
chunks = []
step = chunk_size - overlap
for i in range(0, len(tokens), step):
chunk = detokenize(tokens[i : i + chunk_size])
chunks.append(chunk)
return chunks
The second operational lesson was ingestion throughput. If you process 750 chunks by making 750 individual HTTP calls to an embedding endpoint, your pipeline spends 95% of its time waiting on network roundtrips.
Batching chunks into groups of 256 per request completely reshapes ingestion performance:
- Sequential individual requests: ~25–30 seconds.
- Batched requests (256 / batch): ~3–4 seconds (~8x faster).
The vectors produced are identical, but batching eliminates the HTTP handshake penalty.
How does HNSW solve the scale problem?
Once embeddings are generated, how do you find the closest matches for a user query?
The naive approach is an exhaustive scan (flat search): calculate the cosine distance between the query vector and every single stored vector. This gives 100% recall, but query latency scales linearly () with corpus size. At hundreds of thousands of vectors, latency quickly exceeds interactive thresholds.
Production vector databases solve this with Approximate Nearest Neighbors (ANN) algorithms. The standard approach is Hierarchical Navigable Small World (HNSW).
I visualize HNSW like an interstate highway system:
- Top layers: Sparse graphs with long-distance links. A search takes fast, wide hops across the vector space to locate the general semantic region.
- Bottom layers: Denser graphs with local connections. The search drops down to explore local streets and identify specific nearest neighbors.
HNSW trades a tiny fraction of absolute recall for logarithmic search time (), returning candidates in single-digit milliseconds.
Why cosine similarity is not logical accuracy
A common trap is assuming that because a vector search returned a result with high cosine similarity, that result is logically or factually relevant.
Cosine distance measures the geometric angle between vectors:
A vector database’s top- query is guaranteed to return vectors. If a user asks a question completely outside your documentation’s scope, the index will still happily return the closest items, even if their similarity score is abysmal.
If you blindly pass those chunks into the generation prompt, the LLM hallucinates or gives a plausible-sounding answer based on irrelevant context.
To prevent this, I enforce a strict confidence threshold at the storage layer:
# src/store.py
SIMILARITY_THRESHOLD = 0.70
def query_with_threshold(query_vector: list[float], top_k: int = 5) -> list[dict]:
results = collection.query(query_embeddings=[query_vector], n_results=top_k)
# Cosine distance to similarity: similarity = 1 - distance
filtered_results = [
doc for doc, dist in zip(results["documents"][0], results["distances"][0])
if (1.0 - dist) >= SIMILARITY_THRESHOLD
]
return filtered_results
If no chunk meets the confidence threshold, the pipeline does not call the generator. It immediately triggers a fallback path (“I could not find verified context to answer this question”).
The migration trap: dimensions are immutable
The most painful operational surprise when working with vector databases happens when you decide to change embedding models.
You might think updating your embedding model is a one-line config change:
# DON'T ASSUME THIS WORKS IN PLACE
EMBEDDING_MODEL = "text-embedding-3-large" # 3072 dimensions
It fails immediately. Vector stores like Chroma lock in the collection dimension upon creation: vector schemas are strictly immutable. You cannot insert a 3,072-dimensional vector into an index built for 1,536 floats.
To upgrade an embedding model in production, you have to follow a blue/green migration strategy:
- Spin up a parallel collection (
techdocs_kb_v2). - Run your batched ingestion pipeline to embed the corpus into the new dimension.
- Validate recall and latency on a representative evaluation set.
- Atomically route production query traffic to the new collection.
- Decommission the old collection.
Production takeaways
- Test dimensional impact on domain terms: Before choosing a fast embedder, verify retrieval recall against a sample of specialized terminology.
- Batch ingestion by default: Sending chunks in batches of 128 to 256 reduces ingestion time by an order of magnitude.
- Always gate on a confidence threshold: Never trust raw top- output without checking that the similarity score clears a minimum relevance floor.
- Plan for dual-index migrations: Treat vector collection schemas as immutable; model updates require rebuilding the index in parallel.
Frequently asked questions
Why can't you swap embedding models on an existing collection?
Vector databases lock the vector dimensionality at the very first insert. Changing models requires building a parallel collection from scratch and executing an atomic traffic cutover.
What is the trade-off between exhaustive search and HNSW?
Exhaustive search measures the exact distance to every vector, which scales linearly and degrades at scale. HNSW builds a multi-layered navigable graph, trading a negligible amount of recall for sub-millisecond retrieval.
Why is a confidence threshold mandatory in vector search?
Approximate Nearest Neighbor algorithms always return the top-k closest vectors even if none are relevant. A confidence threshold discards low-similarity noise before it reaches the generation context.