Choosing an embedding model for RAG: a 50-question retrieval test on your own documents

Leaderboards will not tell you which embedding model fits your corpus. Build a 50-question labelled set, score recall@5 for a hosted and a local model, and pick with a rule.

if.codesNov 22, 2026 · 6 min readAI-assisted

New embedding models keep arriving. Cohere released Embed 5 on 30 September 2026 in two tiers, Pro and Fast, and open models you can run through Ollama keep improving. Leaderboards rank them on public datasets, but your documents are not a public dataset: your jargon, your table-heavy PDFs and your short, vague user questions are what decide whether retrieval works. The fix is cheap. Write 50 questions with known answers, score each candidate model on them, and choose with a rule you set before looking at results.

This post gives you the whole harness: a question set format, one Python script that scores recall@5 for a hosted model and a local one, and a decision rule. It needs Python 3, numpy and requests, a running Ollama, and optionally a Cohere API key.

What you are measuring

Evaluating how effectively a model retrieves the correct information chunk.
Evaluating how effectively a model retrieves the correct information chunk.

Retrieval quality for RAG is mostly one question: when the answer exists in your corpus, does the right chunk land in the top few results? That is recall@k. With k = 5 it asks whether the correct chunk appears among the five nearest vectors. We also track latency per query and an estimated cost, because a model that is 2 points better but ten times slower may not be better for you.

Step 1: chunk your documents once

Use the same chunks for every model, otherwise you are testing your chunker. Write them to chunks.jsonl, one JSON object per line:

{"id": "refund-policy-03", "text": "Refunds are issued to the original payment method within 5 business days..."}
{"id": "api-limits-01", "text": "The default rate limit is 60 requests per minute per key..."}

Chunks of a few hundred tokens are a sensible start. Keep the chunk ids stable and human-readable, since you will label questions against them.

Step 2: write 50 questions

Creating a curated set of gold-standard questions and answers.
Creating a curated set of gold-standard questions and answers.

Write questions the way users actually ask them, not copied from the chunk text. Mine support tickets or search logs if you have them. For each, record the id of the chunk that contains the answer in questions.jsonl:

{"question": "how long until I get my money back?", "gold": "refund-policy-03"}
{"question": "what happens if I send too many requests", "gold": "api-limits-01"}

Rules that keep the test honest:

  • Mix easy questions (shares words with the chunk) and hard ones (paraphrase, abbreviation, typo).
  • One gold chunk per question. If two chunks answer it, split the question or pick the better one.
  • Write at least 10 questions you expect your current setup to fail on.
  • Do not let the model, or anyone who has seen the results, edit the questions afterwards.

Fifty questions is small. One question is 2 points of recall, so differences of a few points are noise. That is why the decision rule below uses a margin.

Step 3: the scoring script

The script embeds all chunks and all questions with each candidate model, ranks chunks by cosine similarity, and prints recall@5, mean reciprocal rank, average query latency and an estimated indexing cost. Ollama's /api/embed endpoint takes a model and a list of inputs and returns an embeddings array. Cohere's v2 embed endpoint takes texts, a model, an input_type and embedding_types. Check the exact Cohere model ID in Cohere's documentation before you run it; the ID below is read from an environment variable so you can change it without editing the file.

import json
import os
import time

import numpy as np
import requests

OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434")
COHERE_KEY = os.environ.get("COHERE_API_KEY")
COHERE_MODEL = os.environ.get("COHERE_MODEL", "embed-v5.0-pro")
K = 5


def load_jsonl(path):
    with open(path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]


def embed_ollama(model, texts, prefix=""):
    out = []
    for i in range(0, len(texts), 32):
        batch = [prefix + t for t in texts[i:i + 32]]
        r = requests.post(
            f"{OLLAMA_URL}/api/embed",
            json={"model": model, "input": batch},
            timeout=600,
        )
        r.raise_for_status()
        out.extend(r.json()["embeddings"])
    return np.array(out, dtype=np.float32)


def embed_cohere(model, texts, input_type):
    out = []
    for i in range(0, len(texts), 96):
        r = requests.post(
            "https://api.cohere.com/v2/embed",
            headers={"Authorization": f"Bearer {COHERE_KEY}"},
            json={
                "model": model,
                "texts": texts[i:i + 96],
                "input_type": input_type,
                "embedding_types": ["float"],
            },
            timeout=120,
        )
        r.raise_for_status()
        out.extend(r.json()["embeddings"]["float"])
    return np.array(out, dtype=np.float32)


def normalize(m):
    return m / np.linalg.norm(m, axis=1, keepdims=True)


# Each candidate: how to embed documents and queries.
# nomic-embed-text expects task prefixes; check the model card for yours.
CANDIDATES = {
    "ollama:nomic-embed-text": {
        "doc": lambda t: embed_ollama("nomic-embed-text", t, "search_document: "),
        "query": lambda t: embed_ollama("nomic-embed-text", t, "search_query: "),
        "usd_per_million_tokens": 0.0,
    },
    "ollama:bge-m3": {
        "doc": lambda t: embed_ollama("bge-m3", t),
        "query": lambda t: embed_ollama("bge-m3", t),
        "usd_per_million_tokens": 0.0,
    },
}
if COHERE_KEY:
    CANDIDATES[f"cohere:{COHERE_MODEL}"] = {
        "doc": lambda t: embed_cohere(COHERE_MODEL, t, "search_document"),
        "query": lambda t: embed_cohere(COHERE_MODEL, t, "search_query"),
        # Fill in from Cohere's current pricing page before trusting the cost column.
        "usd_per_million_tokens": float(os.environ.get("COHERE_USD_PER_M", "0")),
    }


def main():
    chunks = load_jsonl("chunks.jsonl")
    questions = load_jsonl("questions.jsonl")
    ids = [c["id"] for c in chunks]
    texts = [c["text"] for c in chunks]
    approx_tokens = sum(len(t) for t in texts) / 4  # rough: 4 characters per token

    print(f"{len(chunks)} chunks, {len(questions)} questions")
    print(f"{'model':32} {'recall@5':>9} {'MRR':>6} {'ms/query':>9} {'index $':>8}")

    for name, c in CANDIDATES.items():
        d = normalize(c["doc"](texts))

        hits, rr, lat = 0, 0.0, []
        for q in questions:
            t0 = time.perf_counter()
            qv = normalize(c["query"]([q["question"]]))[0]
            lat.append((time.perf_counter() - t0) * 1000)
            order = np.argsort(-(d @ qv))
            ranked = [ids[i] for i in order]
            if q["gold"] in ranked[:K]:
                hits += 1
            if q["gold"] in ranked:
                rr += 1.0 / (ranked.index(q["gold"]) + 1)

        n = len(questions)
        cost = approx_tokens / 1e6 * c["usd_per_million_tokens"]
        print(f"{name:32} {hits / n:9.2f} {rr / n:6.2f} {np.median(lat):9.0f} {cost:8.4f}")


if __name__ == "__main__":
    main()

Pull the local models and run it:

ollama pull nomic-embed-text
ollama pull bge-m3
export COHERE_API_KEY=your-key   # optional
export COHERE_MODEL=embed-v5.0-pro   # confirm the ID in Cohere's docs
export COHERE_USD_PER_M=0.12         # copy from Cohere's current pricing page
python score.py

The output is a table like the one below. The numbers here are placeholders to show the format, not measurements; yours will differ.

model                          recall@5    MRR  ms/query  index $
ollama:nomic-embed-text            x.xx   x.xx       xxx   0.0000
ollama:bge-m3                      x.xx   x.xx       xxx   0.0000
cohere:embed-v5.0-pro              x.xx   x.xx       xxx   x.xxxx
Want semantic search like this on your own data?I build RAG and search systems end to end. The estimate is free.

Things that quietly skew the result

  • Task prefixes. Some models, such as nomic-embed-text, are trained with prefixes like search_document: and search_query:. Leaving them out can understate the model. Read each model card.
  • Context length. Check the context window your runtime actually uses. One guide notes the Ollama card for nomic-embed-text lists a smaller context than the model's native window, so long chunks may be truncated. Truncation looks like a bad model.
  • Hosted tiers. Cohere describes Embed 5 Pro as tuned for retrieval quality and Fast as tuned for latency and throughput, and says the two share a compatible vector space, so you can index with Pro and query with Fast. Test that pairing too: it can change your latency-and-cost trade-off without rebuilding the index. Verify this in the Cohere docs for the model you deploy.
  • Latency location. A hosted model's latency includes the network from where your code runs. Measure from the machine that will serve queries, and at the same time of day you expect traffic.

Step 4: a decision rule, written before you run

Weighing performance gains against latency and operational costs.
Weighing performance gains against latency and operational costs.

Write this down first so the results do not pick the rule:

  1. Discard any model whose recall@5 is more than 4 points (two questions) below the best.
  2. Among the rest, choose the cheapest to run at your expected query volume.
  3. If two are within the margin on quality and cost, choose the one with lower median query latency.
  4. If all are within the margin on recall, your retrieval problem is probably chunking or the questions, not the model. Look at the failed questions before buying anything.

Then read the failures. For each question the winner missed, print the gold chunk and the top three chunks it returned. Most misses are chunk boundaries or missing context in the chunk, not embedding quality.

When to re-run

Keep chunks.jsonl, questions.jsonl and the script in your repo. When a new model arrives, add one entry to CANDIDATES and rerun. Re-run too when your corpus changes a lot. The whole run takes minutes, and it replaces an argument about leaderboards with a table of your own data.

Sources

if.codesI build RAG, AI integrations and agent pipelines on Go and Python backends — and write about it here.
// keep reading

More posts on AI and backends.

// free quote

Read something you need? I’ll quote it for free.

RAG, AI integrations, agents or the backend underneath — tell me what you have and what should change. I read every request myself.

Free · no commitment

Tell me what you have. I’ll tell you what it takes.

1Describe the projectA few sentences is enough — about two minutes.
2I review itI read it myself and may ask a follow-up question.
3You get a free quoteScope, approach and estimate — yours to keep, no strings.
What kind of project is it?
Free and without obligation. Your details are used only to reply — see the privacy policy.