System overview · Retrieval pipeline

How Yaqeen Search works

Ask a question in plain language; get back the exact moments in Yaqeen's video library and the passages in its written works that answer it — a video plays at that second, an article opens at its page. Here is what happens between the question and the answer.

2,190

videos indexed

~70k

timestamped moments

1,002

written works, incl. 15 full books

<2s

to results

A Building the library

Done once per video, ahead of time.

Every video's transcript comes from Yaqeen's transcription service already carrying sentence-level timestamps, so we never have to align text to audio — the timing is free. We cut each transcript into ~60-second windows that overlap by 15 seconds (so a thought split across a boundary is still findable), and store every window as a searchable record in GoodMem, our vector database. Each record keeps the text plus its source: video_id, title, and start/end seconds — everything needed to build a jump-to-the-moment YouTube link later.

Two video collections are built: one of the ~70,000 transcript moments, and a second of the 2,190 video titles. The title collection is what lets a strong title match ("Malcolm X's Legacy") lift a whole video, not just one line of it.

A third collection holds the written library — 1,002 papers, blogs, curricula, topic pages and the full text of 15 books, pulled from yaqeeninstitute.org. Each work is stored whole and GoodMem splits it server-side into ~512-character passages (64 overlapping) that each remember their article's title, kind and URL — so a hit deep inside a 400-page book still links to the right page.

B Answering a question

The live path, every search, in well under two seconds. One search box, three tabs: All (the default) runs the video and written paths in parallel; Videos and Written content run one of them.

The question "why is there so much suffering"

Spelling · corpus Query rewrite — words the library never uses become the spelling it does use istekhara → istikhara, omer suleiman → omar suleiman. Built from 90 days of real searches; a word is replaced only when it passes both a spelling and a phonetic threshold against the library's own spelling, and a spelling the library uses even once is left alone. The page says "searched as …" when anything changed.

Semantic · body Transcript moments Embedder finds candidates → Voyage reranker scores each for real relevance

Semantic · title Video titles Same path — how well the video is about the question

Semantic · written Article passages Same embed → rerank path over the written library; passages of one work collapse into one card

Exact · direct Keyword index Any moment or work containing the (rewritten) query as a phrase, verbatim and in order — regardless of the model

Blend Weighted fusion — 0.6 × title relevance + 0.4 × moment relevance A strong title match lifts the whole video; a title that clears the gate is a card by itself. Direct keyword hits are injected here and flagged. Articles carry their best passage's relevance and are interleaved with videos on the All tab by that single number.

relevance gate — keep if score ≥ 0.50 · exact matches exempt

Tidy up Merge & de-duplicate → one card per video, one per article Adjacent moments combine into a single time range; a title-driven video shows as "full video"; an article's extra matching passages are counted, not repeated. Direct matches rank first, then by score. On the All tab, three or more adjacent videos fold into a carousel.

ResultsResults appear instantly Each links to youtube…&t= the exact second

C Why it's built this way

Five decisions that carry the quality of the results.

Piece	What it buys us
Reranker	A raw vector match is fuzzy. The Voyage reranker re-reads each candidate against the actual question and returns a calibrated 0–1 relevance, which is what makes a meaningful cut-off possible at all.
Title weighting	People often search for what a video is about, not a passing line. Weighting the title 60% surfaces the right video as a whole, then points inside it.
Keyword index	Semantic search has a blind spot: names. "Osama bin Laden" pulls the companion "Usama ibn Zayd" instead. A direct keyword match is a certainty the model shouldn't override — so it bypasses the reranker and the threshold. Measured without it, the search lost 12 of 200 good first results, so it stays.
Query rewrite	People spell Arabic words a dozen ways (istikhara, istekhara, isthikhara, istikharah…) and the embedder has only seen the library's spelling. Rewriting to that spelling before retrieval fixes the miss at the source. It is deliberately cautious: both a spelling and a phonetic threshold must pass, and anything the library already spells that way is never touched.
Relevance gate	Vector search always returns something. The 0.50 floor lets the system say "we don't cover this" — an empty result — instead of showing confident-looking but off-topic cards. It is a deliberate precision-first choice, set from a sweep over the evaluators' 142 queries: below 0.40 the gate stops rejecting off-topic questions ("how do I boil eggs") while recall no longer improves.
Whole works, auto-chunked	Written works are stored as one record each and chunked by GoodMem, not by us — so the same passages can later serve as grounded context for an LLM, and a book with twelve matching passages still shows as one card.

D A worked example

The query rewrite earning its place — a query the model alone gets wrong.

query istekhara dua for marrige

Neither istekhara nor marrige occurs anywhere in the library, so the embedder has nothing to anchor on and the reranker scores every candidate low.

In the search log, istekhara sits in a cluster of 33 spellings of the same word, whose label — the spelling the library uses, 125 times — is istikhara. It passes the spelling check (0.89) and the phonetic check (identical sound key).

✓

The search runs as istikhara du'a for marriage and returns the istikhara videos and the marriage du'a pages; the status line says searched as "istikhara du'a for marriage".

Known limits, being honest: the rewrite only knows spellings that appeared in 90 days of searches, and only replaces a word when it is sure — so an unseen misspelling is still searched as typed. A name the library has never mentioned at all cannot be found either way; the thresholds are set so such a name is left as typed rather than turned into a common word. And exact matching is literal: a transcript that mentions a surah in passing can outrank a work about it.

E The stack, at a glance

Layer	What we use
Vector store	GoodMem — three collections (moments, titles, written works); runs semantic search + reranking
Embeddings	Voyage `voyage-4-large`
Reranker	Voyage `rerank-3` — rescores a pool of 50 candidates per collection per search; every response reports whether reranking was actually applied
Chunking	videos: 60-second windows, 15-second overlap · written: GoodMem default, 512 characters with 64 overlap
Tuning	title 0.6 / body 0.4 · relevance floor 0.50 · exact-match override · query rewrite (first letter kept, spelling ≥ 0.80, sound key within 1 letter) · top 10 cards per search

Snapshot of the current configuration — every weight and threshold above is a single tunable value, adjusted from live query traces.