Benchmarks, losses included
We ran the same queries and URLs through HushSearch and the APIs people compare us with, at the same time, and publish every result. Exa beats us on search relevance; we say so.
- Run
- September 27, 2026
- Tag
- phase0-honest-unseeded
- Sample
- 24 searches · 7 pages · 3 answers
Head-to-head records
Search relevance vs Tavily
16–4–4
wins · losses · ties
24 queries
Search relevance vs Exa
5–14–5
wins · losses · ties
24 queries
Scrape cleanliness vs Firecrawl
5–0–2
wins · losses · ties
7 URLs
Scrape cleanliness vs Jina Reader
7–0–0
wins · losses · ties
7 URLs
Answer quality vs Tavily
3–0–0
wins · losses · ties
3 questions
Search relevance
Mean nDCG over 24 queries against graded expected URLs. Higher is better; each expected URL counts once and scores are capped at 1.
- Exa0.51
- HushSearch0.44
- Tavily0.25
Bars are scaled to the best score. Exa leads because it runs its own neural web index; HushSearch uses metasearch and local reranking.
Latency
Wall-clock time per call from the same machine. p95 is the slow tail: ours is the worst on search.
| Task | Provider | OK | p50 | p95 |
|---|---|---|---|---|
| Search | HushSearch | 24/24 | 895 ms | 3,572 ms |
| Search | Tavily | 24/24 | 1,163 ms | 2,260 ms |
| Search | Exa | 24/24 | 1,253 ms | 1,894 ms |
| Scrape | Firecrawl | 7/7 | 399 ms | 454 ms |
| Scrape | HushSearch | 7/7 | 503 ms | 1,218 ms |
| Scrape | Jina Reader | 7/7 | 673 ms | 1,859 ms |
| Answer | HushSearch | 3/3 | 948 ms | 1,016 ms |
| Answer | Tavily | 3/3 | 1,753 ms | 2,029 ms |
Every search query
nDCG per query (0 to 1). Queries span developer docs, research papers, news, shopping, legal and medical topics. Medical queries are our weakest area.
| Query | HushSearch | Tavily | Exa | vs Tavily | vs Exa |
|---|---|---|---|---|---|
| python asyncio tutorial | 0.94 | 0.77 | 0.92 | Win | Tie |
| fastapi authentication JWT | 0.43 | 0.00 | 0.51 | Win | Loss |
| redis caching strategies | 0.52 | 0.26 | 0.52 | Win | Tie |
| postgresql index optimization | 0.21 | 0.00 | 0.49 | Win | Loss |
| attention is all you need transformer paper | 0.28 | 0.32 | 0.79 | Tie | Loss |
| BERT language model pretraining | 0.28 | 0.00 | 0.50 | Win | Loss |
| CRISPR gene editing mechanism review | 0.70 | 0.80 | 0.54 | Loss | Win |
| diffusion models image generation survey | 0.67 | 0.34 | 0.42 | Win | Win |
| latest federal reserve interest rate decision | 0.48 | 0.13 | 0.54 | Win | Loss |
| latest AI regulation news European Union | 0.79 | 0.23 | 0.20 | Win | Win |
| recent SpaceX Starship launch update | 0.56 | 0.52 | 0.57 | Tie | Tie |
| today stock market S&P 500 movement | 0.75 | 0.00 | 0.19 | Win | Win |
| iphone 15 pro review | 0.24 | 0.00 | 0.34 | Win | Loss |
| best mechanical keyboard 2024 | 0.86 | 0.10 | 0.00 | Win | Win |
| sony wh-1000xm5 vs bose quietcomfort | 0.52 | 0.33 | 0.75 | Win | Loss |
| dyson v15 vacuum specifications | 0.60 | 0.65 | 0.65 | Tie | Tie |
| GDPR data subject access request requirements | 0.52 | 0.00 | 0.52 | Win | Tie |
| California CCPA consumer privacy rights | 0.33 | 0.26 | 0.56 | Win | Loss |
| DMCA safe harbor takedown procedure | 0.15 | 0.23 | 0.54 | Loss | Loss |
| software license MIT vs Apache 2.0 differences | 0.11 | 0.00 | 0.64 | Win | Loss |
| type 2 diabetes management guidelines | 0.48 | 0.19 | 0.56 | Win | Loss |
| hypertension blood pressure treatment options | 0.00 | 0.20 | 0.27 | Loss | Loss |
| covid-19 long term symptoms research | 0.11 | 0.13 | 0.25 | Tie | Loss |
| migraine causes and prevention | 0.00 | 0.52 | 0.93 | Loss | Loss |
Method and limits
- One HushSearch instance on a single local machine against the vendors' public APIs, run at the same time. HushSearch had no seeded or hand-picked results.
- Relevance is nDCG against expected URLs graded per query; each expected URL is credited once and no score exceeds 1.
- Scrape cleanliness penalises leaked HTML, script and navigation markup and very short output. It does not measure completeness.
- Answer quality compares citation count and groundedness on 3 questions. That is a small sample; treat it as directional.
- One run, one region, one machine. Hosted latency will differ; this page is not a service-level promise.
- An earlier run that used seeded results was withdrawn and is not shown.
Check it yourself
Self-hosters can re-run the full evaluation with their own vendor keys:
python3 scripts/head-to-head-eval.py --write-report
Or compare one query live in the playground with your key.