Replacing keyword search on the internal wiki
In one line
Vector search fixes the synonym problem and quietly breaks exact lookups; the answer is not to pick one, because each misses in a direction the other catches.
The situation
The internal wiki's search is keyword-based, and everyone hates it. "Reset password" finds the page; "forgot my login" finds nothing, because the page never uses the word "login". A proposal lands: replace it with vector search.
Two weeks after the switch, a new complaint: searching for the error code
E4012 returns five pages about error handling in general and not the one
page that mentions E4012. The old search found it instantly. The team has
traded one class of failure for another, and the interesting question is why
both were predictable.
Two searches, two blind spots#
Keyword search — is the standard scoring — matches on the tokens present. It is precise, fast and completely literal. It cannot know that "login" and "password" are related, so paraphrase defeats it. But an exact, rare token like an error code, a surname or a version number is exactly what it is best at: a term that appears in one document out of ten thousand scores enormously, because rarity is the signal.
Vector search compares s. "Forgot my login" and "reset password"
land close together because the model has seen those phrasings in the same
contexts a million times. But E4012 has never appeared in any meaningful
context; its vector is built from meaningless fragments,
and it sits nowhere in particular. Vector search is excellent at meaning and
nearly blind to identity.
Neither is broken. They fail in orthogonal directions, and a search that only uses one will always have a class of embarrassing miss.
Run both, merge the rankings#
runs both queries and combines the result
lists — commonly with reciprocal rank fusion, which rewards a document for
ranking well in either list without needing the two score scales to agree. The
cost is one extra query. The gain is that "forgot my login" and E4012 both
work.
There is a habit worth building here that generalises well past search: measure against a labelled set. Take two hundred real queries, mark which page each should find, and record how often each approach finds it in the top five. Keyword-only, vector-only and hybrid produce three numbers, and the decision stops being a matter of taste. It also gives you the number to watch when the wiki grows or the embedding model changes.
Questions to ask
- What fraction of my queries contain an exact identifier? Those need keywords.
- What fraction are paraphrases of the page's own words? Those need vectors.
- Do I have two hundred labelled queries to measure recall against?
- When a search misses, can I tell which of the two searches should have caught it?
Mindset
Vector search does not replace keyword search; it covers the gap keyword search has, and opens a gap of its own. Assume both, merge, and measure — the instinct to pick a winner is the mistake.
Where it connects
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.