Tools

ScrapingBee

HTML API scraping; the api_key comes from the gateway

View as Markdown

Fronts the HTML API of ScrapingBee; the gateway fills in the secret api_key parameter on the query string:

GET|POST /scrapingbee   → https://app.scrapingbee.com/api/v1/?api_key=<secret>&<your query>
  • A GET fetches the target URL.
  • A POST is passed on to the target as a POST.

Everything is driven by the query string: the target url plus any ScrapingBee parameter (premium_proxy, render_js, country_code, extract_rules, …). What you put there reaches ScrapingBee exactly as written.

Credential handling

Three auth channels exist on ScrapingBee’s side; the gateway fills one channel and closes the remaining two:

  • api_key goes in ahead of everything you send, so it can’t be swallowed by a target URL with incomplete encoding.
  • An api_key supplied by the client in the query is discarded (any encoding).
  • Authorization is removed — it doubles as an auth channel on ScrapingBee’s side, and it carries your gateway token.
  • The header form of the key, X-API-KEY, is discarded.

Examples

$curl "$GATEWAY/scrapingbee?url=https://example.com" \
>  -H "Authorization: Bearer $TOKEN"

ScrapingBee’s response comes back untouched. Every supported parameter is listed in ScrapingBee’s documentation.