300ms Semantic Search on a Candidate Listing Page
Vector search over hundreds of thousands of candidates is fast, a steady 10 to 30ms. The latency that threatens a search page lives in everything around it. Here's how the whole path holds about 300ms at p95.
This is the candidate browse page recruiters live in: type a few words, get the people who actually fit, ranked by what their profile means rather than which keywords it happens to contain. It replaced a five-year-old ranking built on layered SQL rules that were slow to run and barely relevant.
The vector search at the middle of it is fast, and that part isn’t the interesting one. Comparing a query against hundreds of thousands of candidate vectors runs in 10 to 30 milliseconds, because that is what an approximate-nearest-neighbour index does. Search-by-meaning being quick is something you get off the shelf.
The latency that actually threatens the page lives around that query:
- turning the recruiter’s words into a vector is a network call
- every rephrase or page turn makes a naive build do all of it again
Left alone, those push p95 well past 300 milliseconds.
Holding the whole path around 300ms was mostly about doing each of those things at most once.
What the page actually costs
embed the query → top-K vector search → hydrate the page
Between hitting enter and seeing a page, three steps carry a running cost, and they don’t cost anything like the same. The query becomes a vector through text-embedding-3-small (1536 dimensions, cheap and fast, plenty of quality at this scale). That vector finds its top-K nearest matches, then hydration draws the page by fetching the display details for the rows that will show.
The chart is lopsided on purpose. The embedding call is a network hop out to OpenAI, and its tail is the whole problem: mostly quick, but spiking to half a second and occasionally into whole seconds. Search and hydration are steady and cheap next to that.
Two things stand out. The embedding hop is the largest and least predictable cost, and most of the total is work that repeats across searches that barely change. Both are things you can stop paying for.
The embedding call
same query text → same vector → serve it from cache
Turning text into a vector is the one step that leaves the building. Almost everyone reaches for a hosted embedding API for it, and it’s the right default: one call, no model to run, no GPU to keep alive. The price is that the call carries a network tail, and because teams pick the API for convenience rather than latency, that tail usually goes unmeasured.
You can avoid the hop by running a small model in your own process with something like ONNX Runtime, but a model light enough to be fast on CPU is weaker than text-embedding-3-small, and matching that quality wants a GPU to own and operate. At this scale the trade wasn’t worth it.
The cheaper move is that the same text always produces the same vector, so the vector gets cached on a hash of the query. The first search for a phrase pays the call once; every repeat within the window reads it from Redis.
That helps more than a cache usually does, because it trims the spikiest step: shaving something usually 40ms but sometimes two seconds does far more for p95 than shaving something always fast. And recruiters repeat constantly, tweaking a word, paging, returning to the same query an hour later.
The cache key
Both caches key on an FNV-1a hash of the exact text, a non-cryptographic 64-bit hash that’s fast and stable, which is all “same input, same key” ever needs.
Not searching the same thing twice
rank once → cache ~200 IDs for 5 min → page and re-query from cache
Caching the embedding skips one call. Caching the result skips the search entirely.
The ranked list, cached
The ranker returns an ordered list of about 200 candidate IDs, oversampled on purpose (the search section explains why), and held in Redis for five minutes. Paging from results 1-10 to 11-20, or running the same query again moments later, reads that list from cache: no embedding call and no vector search, because both already ran once.
Hydrate only the page
The cached list is just IDs in ranked order. Hydration fetches the display details for only the ten on screen, never the whole 200. The one catch is order: a relational fetch won’t return rows in the ranker’s order, so that order has to be re-applied as they come back.
Keeping the search on its fast path
approximate nearest-neighbour over HNSW → oversample past the filter → 10-30ms
HNSW is what makes the search quick, walking a graph toward the nearest vectors while visiting only a fraction of them, at about 99% recall.
CREATE INDEX idx_embeddings_hnsw
ON embeddings
USING hnsw (embedding_vector vector_cosine_ops)
WITH (m = 16, ef_construction = 200);
What actually needed care was the filter. A search is never just “find the closest vectors,” it’s “find the closest vectors this recruiter is allowed to see.” The index picks the nearest by distance first, and only then does the eligibility filter run, so a selective filter can throw away most of what came back, leaving fewer than a page, sometimes nothing at all.
The fix is to over-fetch on purpose: ask the index for about twice the candidates a page needs, so that after the filter drops everyone ineligible, a full page of eligible matches still remains.
A relevance cutoff rides along too: matches below a minimum cosine similarity are dropped, so the page never fills with candidates that are only coincidentally near the query.
Two settings are fixed at build time. m (neighbours per node) stays at the default 16, plenty for this dataset and recall target; ef_construction is pushed from 64 to 200, effort spent once for a denser graph every query then rides for free.
The live dial is ef_search, how wide each query looks. It trades recall for latency almost linearly, so you set it against a p95 target and hold recall near 99% inside the budget.
The whole path, and the 300ms
query → embed (cached) → search (HNSW) → result cache → hydrate page
Put together, the page holds its budget by doing each expensive thing at most once.
- p95 end to end is about 300ms, and that’s what the caches make the common case, not the price of a cold search.
- A cold search (empty cache, never-seen query) pays the full embedding hop and can spike past 300ms on its own; the caches make that the rare path.
- Paging and repeat queries skip embedding and search entirely, served from the five-minute result cache.
What the speed costs
None of this made the search algorithm faster. It made the page repeat itself less, and paid for that in freshness: a result set is frozen for five minutes, so a candidate who edits their profile mid-window keeps their old ranking until the cache expires.
On a browse page that trade is easy: five minutes of staleness nobody notices, for a page that answers in about 300ms. That’s the real shape of making something fast, deciding what you’re allowed to not recompute yet.
The embeddings this page searches are kept current by a separate sync pipeline, which has its own cost problem to solve. Here’s how that one stays cheap.
How are you keeping a search page fast, caching the embedding, caching the result set, or something else? I haven’t needed to leave the hosted embedding API yet, and I’m curious where other people drew that line. Let’s talk on LinkedIn.