Tools

ScraperAPI

General-purpose scraping; the gateway supplies the api_key

View as Markdown

Fronts the synchronous endpoint of ScraperAPI; the gateway fills in the secret api_key parameter on the query string.

GET|POST|PUT /scraperapi   → https://api.scraperapi.com/?api_key=<secret>&<your query>
  • GET retrieves the page at url.
  • A POST or PUT delivers a form or body to the target, with the body relayed as-is.

Everything is driven by the query string: the target url plus any ScraperAPI parameter (country_code, render, device_type, premium, …). What you put there reaches ScraperAPI exactly as written.

Credential handling

  • api_key goes in ahead of everything you send — ScraperAPI wants its parameters placed before url, and this also keeps a target URL with incomplete encoding from swallowing them.
  • An api_key supplied by the client in the query is discarded (any encoding), so nothing can displace the injected key.
  • The header form of the key, x-sapi-api_key, is discarded.
  • Authorization is removed, and here that matters beyond hygiene: under keep_headers=true ScraperAPI passes your headers on to the scraped site — a surviving gateway token would therefore leak to a third party.

Every other header — the x-sapi-* controls such as x-sapi-render included — goes through untouched.

Examples

$curl "$GATEWAY/scraperapi?url=https://example.com" \
>  -H "Authorization: Bearer $TOKEN"

Expect slow scrapes — ~70 seconds is the allowance ScraperAPI suggests. Since the gateway simply waits on the upstream, the timeout that actually binds is your client’s. Payloads can be big too (up to ~50MB on ScraperAPI’s side), and the gateway streams every byte back.

ScraperAPI’s response comes back untouched. Every supported parameter is listed in ScraperAPI’s documentation.