Tools

Crawlbase

Crawling API access where the route decides which token gets injected
View as Markdown

Fronts the Crawling API of Crawlbase; the gateway fills in the secret token parameter on the query string:

GET|POST /crawlbase      → https://api.crawlbase.com/?token=<normal>&<your query>
GET|POST /crawlbase/js   → https://api.crawlbase.com/?token=<javascript>&<your query>

Crawlbase hands out two tokens, and which one gets injected follows from the route:

RouteTokenBehavior
/crawlbaseNormalFetches static HTML
/crawlbase/jsJavaScriptLoads the page in a real headless browser

Everything is driven by the query string: the target url plus any parameter Crawlbase defines (country, format, page_wait, …). What you put there reaches Crawlbase exactly as written.

Credential handling

  • token goes in ahead of everything you send, so a target URL with incomplete encoding can’t swallow it. Which token is chosen depends on which route you called — /crawlbase or /crawlbase/js.
  • A token supplied by the client in the query is discarded (any encoding), so nothing can displace the injected one.
  • Authorization is removed before the request leaves the gateway; Crawlbase does nothing with your gateway token.

Examples

$curl "$GATEWAY/crawlbase?url=https://example.com" \
>  -H "Authorization: Bearer $TOKEN"

Crawlbase’s response comes back untouched. Every supported parameter is listed in Crawlbase’s documentation.