Tools

Spider

Scrape, crawl, search, and the rest of Spider's actions on one gateway route
View as Markdown

Fronts the action endpoints of Spider at https://api.spider.cloud: scrape, search, crawl, links, transform, screenshot, and unblocker. The path following /spider/ is relayed upstream, restricted to those actions:

POST /spider/<api path>   → https://api.spider.cloud/<api path>

Admission is default-deny against a fixed allowlist — the action endpoints below are what passes, and any other path (the /data/* storage endpoints included) draws a 404 until the endpoint concerned is added to the allowlist. Mounted methods: POST.

Common paths

PathPurpose
POST /spider/scrapeScrape a URL
POST /spider/crawlCrawl a site
POST /spider/searchSearch the web
POST /spider/linksExtract links
POST /spider/screenshotCapture a screenshot
POST /spider/transformTransform HTML into markdown/clean content
POST /spider/unblockerFetch a page through the unblocker

Credential handling

Spider reads its key from Authorization: Bearer — the very header carrying your gateway token — so rather than stripping that header, the gateway replaces it with the Spider key:

  • Authorization is overwritten with Spider’s secret key. Your gateway token stops at the gateway and never travels upstream.
  • No query-param form of the key exists on Spider’s side, so nothing gets removed from the query string.

Examples

$curl -X POST "$GATEWAY/spider/scrape" \
>  -H "Authorization: Bearer $TOKEN" \
>  -H "Content-Type: application/json" \
>  -d '{"url": "https://example.com"}'

Responses arrive as Spider’s JSON, untouched. Field-level request and response detail lives in Spider’s API documentation; the gateway modifies neither.