Never Re-Embed Twice: How an S3 Cache Saved My RAG Pipeline
Published · Last updated · 6 min read
When building semantic search for an internal engineering platform—indexing microservice documentation, incident runbooks, and API specs—most tutorials suggest a simple workflow:
- Read raw documents.
- Split them into chunks.
- Call an embedding API (like OpenAI or Vertex AI).
- Insert vectors directly into a vector database (Chroma, Qdrant, Pinecone).
For a prototype, this works. But as our corpus grew to thousands of technical documents, treating the vector database as the sole home for embeddings created serious operational friction.
Every time we wanted to tune an index hyperparameter (like HNSW’s M or ef_construction), migrate environments, or recover from index corruption, we had to re-embed the entire corpus from scratch. That meant re-paying API fees, waiting through 45-minute ingestion jobs, and constantly hitting provider rate limits (429 Too Many Requests).
The realization was simple: Vector generation and vector indexing are two completely different problems.
- Vector generation is slow, rate-limited, network-bound, and costs money per token.
- Index construction (building the HNSW graph in RAM or disk) is fast, local, and purely mathematical.
Coupling them together in one ingestion step was our architectural mistake. Here is how I decoupled them using a cloud object storage ledger.
Figure 1: Decoupling vector generation from vector indexing using an S3 or GCS object storage ledger (Watermark: geekyrahul.in • we4ai).
The problem with direct-to-database ingestion
Consider what happens when you treat your vector database as the only storage for your vectors.
1. Index tuning becomes prohibitively expensive
HNSW graphs balance search latency against recall using construction parameters:
M: Maximum number of bidirectional links per vector node.ef_construction: Size of the dynamic candidate list evaluated during index creation.
If query latency spikes and you need to test whether lowering M from 64 to 16 recovers your p95 latency, you cannot easily re-index existing vectors in place. You must rebuild the collection. If your vectors only exist inside the database, you have to call your embedding model again for every single chunk in your corpus.
2. CI/CD test builds burn API budgets
In our automated testing pipeline, every integration test run spun up an ephemeral vector database to verify retrieval mechanics. Re-embedding 1,000 test chunks on every pull request burned through API quotas and introduced random test failures whenever the embedding provider experienced network jitter.
3. Provider rate limits (HTTP 429) create cascading failures
Embedding thousands of chunks cold inevitably collides with Tokens-Per-Minute (TPM) and Requests-Per-Minute (RPM) rate limits. When a network timeout or 429 occurs on chunk 4,800 out of 5,000, a naive script crashes, leaving the database in an inconsistent, half-populated state.
Decoupling: Cloud object storage as the vector ledger
To solve this, I introduced an externalized vector ledger backed by cloud object storage (Amazon S3 or Google Cloud Storage).
The ingestion pipeline now operates as a two-phase system:
- Phase A (Vector Ledger): Given a text chunk, compute a deterministic hash. If the hash exists in the cloud ledger (
s3://company-rag-cache/embeddings.jsonl), read the vector. If not, call the embedding API and append the new vector to the ledger. - Phase B (Index Hydration): Stream vectors from the cloud ledger directly into the vector database’s HNSW index.
# src/embedder_cache.py
import hashlib
import json
from pathlib import Path
def compute_chunk_key(model_name: str, text: str) -> str:
"""Compound key prevents collisions across model updates."""
payload = f"{model_name}:{text}".encode("utf-8")
return hashlib.sha256(payload).hexdigest()
def get_or_embed(chunks: list[dict], cache: dict[str, list[float]], model: str) -> list[list[float]]:
missing_chunks = []
keys = [compute_chunk_key(model, c["text"]) for c in chunks]
for key, chunk in zip(keys, chunks):
if key not in cache:
missing_chunks.append((key, chunk["text"]))
if missing_chunks:
# Only uncached chunks hit the external API
new_vectors = call_embedding_api([text for _, text in missing_chunks], model=model)
for (key, _), vector in zip(missing_chunks, new_vectors):
cache[key] = vector
append_to_cloud_ledger(key, vector)
return [cache[k] for k in keys]
Why compound hashing is mandatory
A mistake I made early on was keying the cache solely by hashlib.sha256(chunk_text).
This created a subtle bug during an experiment. I switched our embedding model from OpenAI’s text-embedding-3-small (1,536 dimensions) to a local sentence-transformer (384 dimensions). Because the text had not changed, the cache happily returned the cached 1,536-dimensional vectors for the new model, causing immediate runtime assertion crashes when hydrating the local index.
The fix was compound hashing:
If you change the model, the hash changes, and the cache gracefully falls back to generating fresh vectors without corrupting the historical ledger.
Operational benefits observed in production
Decoupling vector generation from the database yielded four immediate operational advantages:
| Operational Metric | Direct-to-Database | Cloud Ledger Architecture |
|---|---|---|
| Cold Rebuild Duration | ~35 to 45 minutes (API latency) | Under 2 seconds (local I/O) |
| Re-Index API Cost | Full corpus re-billing ($$$) | $0.00 (reused existing vectors) |
| Incremental Updates | Full re-scan of collections | Instant SHA-256 diff (only 5 changed pages hit API) |
| CI/CD Test Builds | Flaky on network & burns quotas | 100% offline-capable & deterministic |
| 429 Rate-Limit Recovery | Fails halfway, restart required | Append-only idempotency; resumes at exact failure point |
1. Portable across databases (The zero-cost migration plan)
If tomorrow we need to migrate from Chroma to Qdrant, Pinecone, or Milvus, we don’t need to touch the embedding API. The migration pipeline is clean, fast, and completely free:
[Raw Documents]
│
▼ (Free, pure Python, takes ~1 sec)
[Document Chunker] ────────> Generates fresh: (Doc ID, Text, Metadata)
│
▼ (Check SHA-256 compound hash in cloud cache)
[Cloud Embedding Cache] ───> Hash match! Vector retrieved instantly ($0 API cost)
│
▼
[New DB: Qdrant / Pinecone] ─> Batch upsert(ID, Text, Metadata, Vector)
Because the raw vectors and metadata schemas are preserved in the cloud ledger, migrating to a new vector engine is simply a parallel write without re-paying for token embedding.
2. Painless blue/green deployments
When restructuring our metadata schema (such as adding team ownership tags or service tier filters), we can spin up a new collection in parallel, populate it from the cloud cache in under two seconds, verify search quality, and atomically point our API gateway to the new index.
The evolutionary path: from JSONL to Parquet
For our initial deployment, an append-only JSONL file in cloud storage was the simplest implementation:
- Human-readable.
- Easily appended with standard object write streams.
- Zero dependencies.
However, as a corpus approaches hundreds of thousands of chunks, storing 1,536 floating-point numbers as JSON text causes file bloat (around 15–20 KB per line).
At that scale, transitioning the ledger from JSONL to Apache Parquet or Arrow IPC is the natural step:
- Vectors are stored as native binary
float32arrays, shrinking storage by 70%. - Engines like DuckDB or Polars can query and filter chunks directly in cloud object storage without loading the entire dataset into memory.
Key takeaways
- Never treat your vector database as your primary data store: Databases are indexing engines, not durable storage.
- Store vectors in a cloud object ledger: Keep raw embeddings in S3 or GCS keyed by a compound hash (
model + version + text). - Isolate compute from indexing: Pay the API/GPU token cost once; rebuild and tune your vector graphs as often as you want for free.
- Build for failure: An append-only ledger turns rate-limit crashes and network timeouts into simple resume operations.