# Site Crawl

> Site Crawl API endpoint — 3 credits per call.

Source: http://localhost:3000/docs/endpoints/crawl

## Crawl: `POST /v1/crawl/`

Follow a site's links to a depth and return every page as Markdown. Cost: 3 credits.

```bash
curl -X POST https://api.hushsearch.io/v1/crawl/ \
  -H "X-API-Key: hs_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://books.toscrape.com/",
  "max_depth": 1,
  "max_pages": 5
}'
```

## Crawl Pipeline: `POST /v1/crawl/pipeline`

Map → filter → scrape in one call with include/exclude rules. Cost: 4 credits.

```bash
curl -X POST https://api.hushsearch.io/v1/crawl/pipeline \
  -H "X-API-Key: hs_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://books.toscrape.com/",
  "max_pages": 5,
  "formats": [
    "markdown"
  ]
}'
```

BFS crawl from a starting URL. Follows internal links with configurable depth and page count.

```bash
curl -X POST https://api.hushsearch.io/v1/crawl/ \
  -H "X-API-Key: hs_live_..." \
  -d '{"url":"https://docs.example.com","max_depth":2,"max_pages":10}'
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | Starting URL |
| `max_depth` | integer | 2 | Maximum link depth (BFS) |
| `max_pages` | integer | 10 | Max pages to crawl (≤50) |
| `include_patterns` | string[] | — | Regex: URLs to include |
| `exclude_patterns` | string[] | — | Regex: URLs to skip |
