New embedding models keep arriving. Cohere released Embed 5 on 30 September 2026 in two tiers, Pro and Fast, and open models you can run through Ollama keep improving. Leaderboards rank them on public datasets, but your documents are not a public dataset: your jargon, your table-heavy PDFs and your short, vague user questions are what decide whether retrieval works. The fix is cheap. Write 50 questions with known answers, score each candidate model on them, and choose with a rule you set before looking at results.
This post gives you the whole harness: a question set format, one Python script that scores recall@5 for a hosted model and a local one, and a decision rule. It needs Python 3, numpy and requests, a running Ollama, and optionally a Cohere API key.
What you are measuring

Retrieval quality for RAG is mostly one question: when the answer exists in your corpus, does the right chunk land in the top few results? That is recall@k. With k = 5 it asks whether the correct chunk appears among the five nearest vectors. We also track latency per query and an estimated cost, because a model that is 2 points better but ten times slower may not be better for you.
Step 1: chunk your documents once
Use the same chunks for every model, otherwise you are testing your chunker. Write them to chunks.jsonl, one JSON object per line:
{"id": "refund-policy-03", "text": "Refunds are issued to the original payment method within 5 business days..."}
{"id": "api-limits-01", "text": "The default rate limit is 60 requests per minute per key..."}
Chunks of a few hundred tokens are a sensible start. Keep the chunk ids stable and human-readable, since you will label questions against them.
Step 2: write 50 questions

Write questions the way users actually ask them, not copied from the chunk text. Mine support tickets or search logs if you have them. For each, record the id of the chunk that contains the answer in questions.jsonl:
{"question": "how long until I get my money back?", "gold": "refund-policy-03"}
{"question": "what happens if I send too many requests", "gold": "api-limits-01"}
Rules that keep the test honest:
- Mix easy questions (shares words with the chunk) and hard ones (paraphrase, abbreviation, typo).
- One gold chunk per question. If two chunks answer it, split the question or pick the better one.
- Write at least 10 questions you expect your current setup to fail on.
- Do not let the model, or anyone who has seen the results, edit the questions afterwards.
Fifty questions is small. One question is 2 points of recall, so differences of a few points are noise. That is why the decision rule below uses a margin.
Step 3: the scoring script
The script embeds all chunks and all questions with each candidate model, ranks chunks by cosine similarity, and prints recall@5, mean reciprocal rank, average query latency and an estimated indexing cost. Ollama's /api/embed endpoint takes a model and a list of inputs and returns an embeddings array. Cohere's v2 embed endpoint takes texts, a model, an input_type and embedding_types. Check the exact Cohere model ID in Cohere's documentation before you run it; the ID below is read from an environment variable so you can change it without editing the file.
import json
import os
import time
import numpy as np
import requests
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434")
COHERE_KEY = os.environ.get("COHERE_API_KEY")
COHERE_MODEL = os.environ.get("COHERE_MODEL", "embed-v5.0-pro")
K = 5
def load_jsonl(path):
with open(path, encoding="utf-8") as f:
return [json.loads(line) for line in f if line.strip()]
def embed_ollama(model, texts, prefix=""):
out = []
for i in range(0, len(texts), 32):
batch = [prefix + t for t in texts[i:i + 32]]
r = requests.post(
f"{OLLAMA_URL}/api/embed",
json={"model": model, "input": batch},
timeout=600,
)
r.raise_for_status()
out.extend(r.json()["embeddings"])
return np.array(out, dtype=np.float32)
def embed_cohere(model, texts, input_type):
out = []
for i in range(0, len(texts), 96):
r = requests.post(
"https://api.cohere.com/v2/embed",
headers={"Authorization": f"Bearer {COHERE_KEY}"},
json={
"model": model,
"texts": texts[i:i + 96],
"input_type": input_type,
"embedding_types": ["float"],
},
timeout=120,
)
r.raise_for_status()
out.extend(r.json()["embeddings"]["float"])
return np.array(out, dtype=np.float32)
def normalize(m):
return m / np.linalg.norm(m, axis=1, keepdims=True)
# Each candidate: how to embed documents and queries.
# nomic-embed-text expects task prefixes; check the model card for yours.
CANDIDATES = {
"ollama:nomic-embed-text": {
"doc": lambda t: embed_ollama("nomic-embed-text", t, "search_document: "),
"query": lambda t: embed_ollama("nomic-embed-text", t, "search_query: "),
"usd_per_million_tokens": 0.0,
},
"ollama:bge-m3": {
"doc": lambda t: embed_ollama("bge-m3", t),
"query": lambda t: embed_ollama("bge-m3", t),
"usd_per_million_tokens": 0.0,
},
}
if COHERE_KEY:
CANDIDATES[f"cohere:{COHERE_MODEL}"] = {
"doc": lambda t: embed_cohere(COHERE_MODEL, t, "search_document"),
"query": lambda t: embed_cohere(COHERE_MODEL, t, "search_query"),
# Fill in from Cohere's current pricing page before trusting the cost column.
"usd_per_million_tokens": float(os.environ.get("COHERE_USD_PER_M", "0")),
}
def main():
chunks = load_jsonl("chunks.jsonl")
questions = load_jsonl("questions.jsonl")
ids = [c["id"] for c in chunks]
texts = [c["text"] for c in chunks]
approx_tokens = sum(len(t) for t in texts) / 4 # rough: 4 characters per token
print(f"{len(chunks)} chunks, {len(questions)} questions")
print(f"{'model':32} {'recall@5':>9} {'MRR':>6} {'ms/query':>9} {'index $':>8}")
for name, c in CANDIDATES.items():
d = normalize(c["doc"](texts))
hits, rr, lat = 0, 0.0, []
for q in questions:
t0 = time.perf_counter()
qv = normalize(c["query"]([q["question"]]))[0]
lat.append((time.perf_counter() - t0) * 1000)
order = np.argsort(-(d @ qv))
ranked = [ids[i] for i in order]
if q["gold"] in ranked[:K]:
hits += 1
if q["gold"] in ranked:
rr += 1.0 / (ranked.index(q["gold"]) + 1)
n = len(questions)
cost = approx_tokens / 1e6 * c["usd_per_million_tokens"]
print(f"{name:32} {hits / n:9.2f} {rr / n:6.2f} {np.median(lat):9.0f} {cost:8.4f}")
if __name__ == "__main__":
main()
Pull the local models and run it:
ollama pull nomic-embed-text
ollama pull bge-m3
export COHERE_API_KEY=your-key # optional
export COHERE_MODEL=embed-v5.0-pro # confirm the ID in Cohere's docs
export COHERE_USD_PER_M=0.12 # copy from Cohere's current pricing page
python score.py
The output is a table like the one below. The numbers here are placeholders to show the format, not measurements; yours will differ.
model recall@5 MRR ms/query index $
ollama:nomic-embed-text x.xx x.xx xxx 0.0000
ollama:bge-m3 x.xx x.xx xxx 0.0000
cohere:embed-v5.0-pro x.xx x.xx xxx x.xxxx



