Scrapfly
Fronts the Web Scraping API of Scrapfly; the gateway fills in the secret key parameter on the query string:
Everything is driven by the query string: the target url plus any Scrapfly parameter (asp, render_js, country, proxy_pool, format, …). What you put there reaches Scrapfly exactly as written.
The method of your request becomes the method Scrapfly executes against the scraped URL — hence GET, POST, PUT, PATCH, HEAD all pass through. (OPTIONS is a CORS preflight, not a scrape — the gateway rejects it.)
Credential handling
keygoes in ahead of everything you send, so it can’t be swallowed by a target URL with incomplete encoding.- A
keysupplied by the client in the query is discarded (any encoding), so nothing can displace the injected key. Authorizationis dropped before the request goes upstream — your gateway token is meaningless to Scrapfly.
Examples
Basic scrape
Render JS + anti-scraping protection
Scrapfly’s response comes back untouched. Every supported parameter is listed in Scrapfly’s documentation.
Large responses
When asp renders a page in a browser, Scrapfly’s JSON also carries every XHR request the page made, in result.browser_data. On a Trustpilot review page that was 730 KB of an 814 KB response; the page content was 20 KB and the extraction 19 KB. Through MCP, a response past 512 KB is cut down to fit when it can be (a list keeps its first items, a string its start), each cut listed in omitted.
To receive only what you need, add proxified_response=true: the response is the page content itself, or only the extraction when extraction_model or extraction_prompt is set. For the same page that was 22 KB, billed the same credits.