Vector Databases, Embeddings and Why Semantic Search Broke Keyword Thinking

The explainers on this topic have converged on a standard shape. Embeddings are numerical representations of meaning. Similarity is measured with cosine distance. Indexing uses HNSW. Queries return in under a hundred milliseconds. All correct, all in the better articles already ranking, and none of it tells you why your content did or did not surface.

What is consistently missing is the preprocessing — the part that happens before a single vector exists — plus the reranking stage that happens after, and the operational reality that embedding models get replaced and your entire index becomes incomparable overnight.

Those three omissions are where the practical consequences live.

What an embedding is, without the hand-waving

An embedding model takes text and returns a fixed-length list of numbers — commonly 384, 768, 1,536 or 3,072 of them. The claim is that this list encodes meaning. What it actually encodes is position in a space arranged so that text used in similar contexts during training lands nearby.

That distinction matters. The model has no concept of truth or topic. It has a geometry learned from co-occurrence. Two passages are “similar” if the training data used comparable language around comparable things.

Why cosine, and what it is blind to

Almost every article names cosine similarity. Very few say why, and the why explains a failure mode.

cosine similarity = (A · B) / (‖A‖ ‖B‖)

It measures the ANGLE between two vectors and normalises
away their MAGNITUDE.

Why that is chosen: magnitude in an embedding correlates
heavily with text length and token frequency. A 2,000-word
page and a 40-word answer about the same thing produce
vectors pointing in a similar direction with very different
lengths. Cosine says they match. Euclidean distance says
they are far apart.

Why it costs you: cosine is deliberately blind to
"how much" and only sees "in what direction." A thin page
that gestures at a topic and a comprehensive page that
answers it can sit at nearly the same angle.

Depth is not a property the geometry can see.

Sit with that last line. The retrieval layer, by design, cannot distinguish a superficial treatment from an authoritative one. Everything you believe about comprehensiveness winning is a belief about the ranking layer, not the retrieval layer — and they are separate systems with different objectives.

Dimensionality is a cost decision, not a quality one

Articles mention that embeddings range from a few hundred to a few thousand dimensions and stop there, as though more is better. The trade-off is real and runs in both directions.

	Lower dimensions (384–768)	Higher dimensions (1,536–3,072)
Storage and memory	Substantially cheaper	Four to eight times the footprint
Query latency	Faster	Slower, though index structure matters more
Fine distinctions	Coarser; near-synonyms collapse	Better separation of subtle differences
Behaviour at scale	Degrades sooner as corpus grows	Holds separation longer
Practical effect	Good enough for most narrow corpora	Needed when the corpus is large and topically dense

Chunking: the step nobody writes about and everybody is affected by

Here is the omission that matters most, and I want to be emphatic about it.

Before anything is embedded, documents are cut into pieces. An embedding represents a chunk, not a page. Retrieval returns chunks. The model reads chunks. Your page, as a coherent argued whole, does not exist anywhere in this pipeline after the first step.

Where the cuts fall determines what is retrievable, and the strategies produce meaningfully different outcomes.

Strategy	How it cuts	Failure mode
Fixed token windows	Every N tokens, regardless of content	Cuts mid-argument. Half a conditional is worse than none
Fixed with overlap	N tokens, sliding with shared margin	Better continuity, larger index, duplicate near-matches
Recursive by separator	Splits on paragraph, then sentence, as needed	Respects your structure — if your structure is real
Semantic chunking	Cuts where embedding similarity drops between sentences	Expensive; sensitive to writing style
Structure-aware	Splits on headings and document hierarchy	Best available, and entirely dependent on your markup

The practical test I would apply to any page: take any single section, remove everything around it, and ask whether it still states what it is about and remains true without the surrounding context. If it relies on a pronoun referring back three paragraphs, or on a definition established earlier, it will fail retrieval or produce a misleading citation.

Why pure vector search is not enough

Semantic search has a specific weakness that gets glossed over: it is bad at exact tokens. Product codes, version numbers, surnames, error strings, acronyms. Anything where the precise string is the point.

An embedding of “error TS2345” sits near other error discussions in general. It does not reliably retrieve the one page about that specific code, because the model learned that error codes are a category of thing rather than learning each code individually.

Which is why production systems run hybrid retrieval: a lexical index such as BM25 alongside the vector index, results fused. The lexical side catches exact strings, the vector side catches paraphrase, and a fusion step combines the rankings.

The consequence for anyone producing content: keyword matching did not die. It was demoted to one of two retrieval paths. Exact terminology still matters for anything where the specific string is what someone would type — which is more of the technical and commercial long tail than the “semantic search means keywords are dead” discourse allows.

Reranking, the stage that is always omitted

Every explainer stops at “similar vectors are returned.” Production systems do not stop there, and the missing stage is where a great deal of the actual selection happens.

Real pipeline:

  query
    ↓
  [1] retrieve broadly — vector + lexical, top 50-200 chunks
        fast, approximate, high recall, low precision
    ↓
  [2] rerank — a cross-encoder scores each (query, chunk) PAIR
        slow, accurate, reorders aggressively
    ↓
  [3] select top 3-10 for the context window
    ↓
  generation

Stage 1 asks: is this chunk in the right neighbourhood?
Stage 2 asks: does this chunk actually answer this question?

They are different questions and stage 2 routinely
demotes things stage 1 ranked first.

The distinction matters strategically. A bi-encoder embeds query and document separately and compares — fast, and it never sees them together. A cross-encoder processes the pair jointly and can judge whether the passage genuinely responds to the question. That is why a chunk can be topically adjacent and still get dropped: it was in the neighbourhood but it did not answer.

Content that directly answers a question in its own words outperforms content that merely discusses the topic, and reranking is the mechanism by which that happens. This is the technical justification for the answer-first structure that good writers arrived at by instinct.

Embedding drift, and the migration nobody budgets for

Three consequences worth internalising:

• Version your index. Store which embedding model and version produced every vector. Without it you cannot debug a quality regression and you cannot migrate safely.
• Expect to re-embed periodically. Treat it as maintenance with a cost line, in the same category as reindexing a search engine.
• Your relative position can change with no change to your content. If a system re-embeds its corpus with a newer model, every document moves. You may gain or lose retrievability because the geometry shifted underneath everyone.

That third point deserves emphasis for anyone tracking AI visibility. A discontinuity in your numbers may be an embedding migration at a provider. It is unannounced, unlogged and indistinguishable from your own performance changing.

What this changes about how I write

• Every section stands alone. Self-contained, with its subject named rather than pronouned. Written to be excerpted, because it will be.
• Answer in the first sentence of the section, then elaborate. Reranking rewards the passage that responds directly.
• Use the exact terminology alongside the paraphrase. Hybrid retrieval means you want both paths. Naming the specific version number or error string is not keyword stuffing, it is lexical retrieval insurance.
• Headings as boundaries, not decoration. Under structure-aware chunking they define the units. A heading that does not describe the content beneath it is a mislabelled chunk.
• Stop assuming depth is visible to retrieval. Cosine cannot see it. Depth wins at reranking and with human readers, and it does not help you clear the first gate.

The honest limit

Everything above describes how these systems work in general. You cannot know the specific chunking strategy, embedding model, hybrid weighting or reranker used by any assistant you care about, because none of them publish it and all of them change it.

So the goal is not to optimise for a configuration. It is to write documents that degrade gracefully across the plausible range of configurations — which, conveniently, is also what makes them good for people. Self-contained sections, direct answers, precise terminology and honest structure work under every chunking strategy in the table above and read well to a human.

That convergence is the only genuinely durable thing I have found in this field. Where a technical constraint and good writing point in the same direction, follow both and stop looking for the trick.

The full selection pipeline this feeds into is in how retrieval actually selects your page, and the structural writing consequence has its own treatment in the passage is the unit now, not the page.

Frequently asked questions