Site Crawl
Site Crawl API endpoint — 3 credits per call.
curl -X POST https://api.hushsearch.io/v1/crawl/ \
-H "X-API-Key: hs_live_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://books.toscrape.com/",
"max_depth": 1,
"max_pages": 5
}'Crawl PipelineCrawl & mapLong-running: consider /v1/jobs
POST/v1/crawl/pipeline4 credits
Request
curl -X POST https://api.hushsearch.io/v1/crawl/pipeline \
-H "X-API-Key: hs_live_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://books.toscrape.com/",
"max_pages": 5,
"formats": [
"markdown"
]
}'BFS crawl from a starting URL. Follows internal links with configurable depth and page count.
curl -X POST https://api.hushsearch.io/v1/crawl/ \
-H "X-API-Key: hs_live_..." \
-d '{"url":"https://docs.example.com","max_depth":2,"max_pages":10}'| Parameter | Type | Default | Description |
|---|---|---|---|
url | string | required | Starting URL |
max_depth | integer | 2 | Maximum link depth (BFS) |
max_pages | integer | 10 | Max pages to crawl (≤50) |
include_patterns | string[] | — | Regex: URLs to include |
exclude_patterns | string[] | — | Regex: URLs to skip |