> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nativeport.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nativeport.ai/_mcp/server.

# Spider

Fronts the action endpoints of [Spider](https://spider.cloud) at `https://api.spider.cloud`: `scrape`, `search`, `crawl`, `links`, `transform`, `screenshot`, and `unblocker`. The path following `/spider/` is relayed upstream, restricted to those actions:

```
POST /spider/<api path>   → https://api.spider.cloud/<api path>
```

Admission is default-deny against a fixed allowlist — the action endpoints below are what passes, and any other path (the `/data/*` storage endpoints included) draws a `404` until the endpoint concerned is added to the allowlist. Mounted methods: `POST`.

## Common paths

| Path                      | Purpose                                    |
| ------------------------- | ------------------------------------------ |
| `POST /spider/scrape`     | Scrape a URL                               |
| `POST /spider/crawl`      | Crawl a site                               |
| `POST /spider/search`     | Search the web                             |
| `POST /spider/links`      | Extract links                              |
| `POST /spider/screenshot` | Capture a screenshot                       |
| `POST /spider/transform`  | Transform HTML into markdown/clean content |
| `POST /spider/unblocker`  | Fetch a page through the unblocker         |

## Credential handling

Spider reads its key from `Authorization: Bearer` — the very header carrying your gateway token — so rather than stripping that header, the gateway **replaces** it with the Spider key:

* **`Authorization` is overwritten with Spider's secret key.** Your gateway token stops at the gateway and never travels upstream.
* No query-param form of the key exists on Spider's side, so nothing gets removed from the query string.

## Examples

#### Scrape a URL

```bash
curl -X POST "$GATEWAY/spider/scrape" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

#### Crawl a site

```bash
curl -X POST "$GATEWAY/spider/crawl" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 50}'
```

Responses arrive as Spider's JSON, untouched. Field-level request and response detail lives in [Spider's API documentation](https://spider.cloud/docs/api); the gateway modifies neither.