Benchmarks, losses included

We ran the same queries and URLs through HushSearch and the APIs people compare us with, at the same time, and publish every result. Exa beats us on search relevance; we say so.

Run
September 27, 2026
Tag
phase0-honest-unseeded
Sample
24 searches · 7 pages · 3 answers

Head-to-head records

  • Search relevance vs Tavily

    16–4–4

    wins · losses · ties

    24 queries

  • Search relevance vs Exa

    5–14–5

    wins · losses · ties

    24 queries

  • Scrape cleanliness vs Firecrawl

    5–0–2

    wins · losses · ties

    7 URLs

  • Scrape cleanliness vs Jina Reader

    7–0–0

    wins · losses · ties

    7 URLs

  • Answer quality vs Tavily

    3–0–0

    wins · losses · ties

    3 questions

Search relevance

Mean nDCG over 24 queries against graded expected URLs. Higher is better; each expected URL counts once and scores are capped at 1.

  • Exa0.51
  • HushSearch0.44
  • Tavily0.25

Bars are scaled to the best score. Exa leads because it runs its own neural web index; HushSearch uses metasearch and local reranking.

Latency

Wall-clock time per call from the same machine. p95 is the slow tail: ours is the worst on search.

Latency by task and provider
TaskProviderOKp50p95
SearchHushSearch24/24895 ms3,572 ms
SearchTavily24/241,163 ms2,260 ms
SearchExa24/241,253 ms1,894 ms
ScrapeFirecrawl7/7399 ms454 ms
ScrapeHushSearch7/7503 ms1,218 ms
ScrapeJina Reader7/7673 ms1,859 ms
AnswerHushSearch3/3948 ms1,016 ms
AnswerTavily3/31,753 ms2,029 ms

Every search query

nDCG per query (0 to 1). Queries span developer docs, research papers, news, shopping, legal and medical topics. Medical queries are our weakest area.

Per-query search relevance
QueryHushSearchTavilyExavs Tavilyvs Exa
python asyncio tutorial0.940.770.92WinTie
fastapi authentication JWT0.430.000.51WinLoss
redis caching strategies0.520.260.52WinTie
postgresql index optimization0.210.000.49WinLoss
attention is all you need transformer paper0.280.320.79TieLoss
BERT language model pretraining0.280.000.50WinLoss
CRISPR gene editing mechanism review0.700.800.54LossWin
diffusion models image generation survey0.670.340.42WinWin
latest federal reserve interest rate decision0.480.130.54WinLoss
latest AI regulation news European Union0.790.230.20WinWin
recent SpaceX Starship launch update0.560.520.57TieTie
today stock market S&P 500 movement0.750.000.19WinWin
iphone 15 pro review0.240.000.34WinLoss
best mechanical keyboard 20240.860.100.00WinWin
sony wh-1000xm5 vs bose quietcomfort0.520.330.75WinLoss
dyson v15 vacuum specifications0.600.650.65TieTie
GDPR data subject access request requirements0.520.000.52WinTie
California CCPA consumer privacy rights0.330.260.56WinLoss
DMCA safe harbor takedown procedure0.150.230.54LossLoss
software license MIT vs Apache 2.0 differences0.110.000.64WinLoss
type 2 diabetes management guidelines0.480.190.56WinLoss
hypertension blood pressure treatment options0.000.200.27LossLoss
covid-19 long term symptoms research0.110.130.25TieLoss
migraine causes and prevention0.000.520.93LossLoss

Method and limits

  • One HushSearch instance on a single local machine against the vendors' public APIs, run at the same time. HushSearch had no seeded or hand-picked results.
  • Relevance is nDCG against expected URLs graded per query; each expected URL is credited once and no score exceeds 1.
  • Scrape cleanliness penalises leaked HTML, script and navigation markup and very short output. It does not measure completeness.
  • Answer quality compares citation count and groundedness on 3 questions. That is a small sample; treat it as directional.
  • One run, one region, one machine. Hosted latency will differ; this page is not a service-level promise.
  • An earlier run that used seeded results was withdrawn and is not shown.

Check it yourself

Self-hosters can re-run the full evaluation with their own vendor keys:

python3 scripts/head-to-head-eval.py --write-report

Or compare one query live in the playground with your key.