Tools

Scrapfly

Scraping via Scrapfly with the secret key added by the gateway
View as Markdown

Fronts the Web Scraping API of Scrapfly; the gateway fills in the secret key parameter on the query string:

GET|POST|PUT|PATCH|HEAD /scrapfly   → https://api.scrapfly.io/scrape?key=<secret>&<your query>

Everything is driven by the query string: the target url plus any Scrapfly parameter (asp, render_js, country, proxy_pool, format, …). What you put there reaches Scrapfly exactly as written.

The method of your request becomes the method Scrapfly executes against the scraped URL — hence GET, POST, PUT, PATCH, HEAD all pass through. (OPTIONS is a CORS preflight, not a scrape — the gateway rejects it.)

Credential handling

  • key goes in ahead of everything you send, so it can’t be swallowed by a target URL with incomplete encoding.
  • A key supplied by the client in the query is discarded (any encoding), so nothing can displace the injected key.
  • Authorization is dropped before the request goes upstream — your gateway token is meaningless to Scrapfly.

Examples

$curl "$GATEWAY/scrapfly?url=https://example.com" \
>  -H "Authorization: Bearer $TOKEN"

Scrapfly’s response comes back untouched. Every supported parameter is listed in Scrapfly’s documentation.