Proxy Setup for Scrapy, Playwright and Puppeteer

Each scraping framework handles proxies in its own way, and the differences show up as confusing failures rather than clear errors. This guide covers the practical wiring for Scrapy, Playwright and Puppeteer, the protocol choice that sits underneath all three, and how to tell which layer is actually broken when something stops working.

Choose the protocol before you choose the code

A mobile proxy usually exposes two endpoints for the same connection: an HTTP proxy port and a SOCKS5 port. The HTTP endpoint understands web requests and is the path of least resistance for most libraries, since almost everything supports it and it works cleanly with standard proxy environment variables.

SOCKS5 operates lower down and simply forwards TCP, and it can carry UDP as well where the implementation supports UDP association. Choose it when you need protocols beyond HTTP, when you want DNS resolution to happen at the proxy rather than on your machine, or when your client handles SOCKS more reliably than it handles HTTP CONNECT tunnelling.

The DNS point deserves emphasis. If your client resolves hostnames locally, your lookups reveal your real location and may return a different edge server than a local user would get. For location-sensitive work such as checking regional search results, make sure resolution happens through the proxy. In many libraries that means choosing the remote-DNS variant of the SOCKS scheme explicitly.

• HTTP proxy port: widest compatibility, simplest setup
• SOCKS5 port: lower level, supports UDP, better for remote DNS
• Confirm which port maps to which protocol before writing code
• Decide where DNS resolution should happen and verify it

Scrapy

Scrapy reads a proxy from the meta dictionary on each request, which makes per-request control natural. Set the proxy value on the request meta and the built-in HTTP proxy middleware applies it. Credentials embedded in the proxy URL work, though putting them in a Proxy-Authorization header set by your own middleware keeps them out of logs.

The tidiest pattern is a small downloader middleware that assigns the proxy at request time. That gives you one place to implement rotation policy, one place to attach credentials and one place to react when a response comes back looking blocked. Keep the policy out of your spiders so the spiders stay about parsing.

For SOCKS5 in Scrapy you will need an additional package, since the framework does not speak SOCKS natively. Pair the proxy configuration with sensible values for concurrent requests per domain, Scrapy's AutoThrottle extension and download delay, so your pacing is defined in the same place as your routing.

Playwright

Playwright accepts a proxy configuration at browser launch, and separately at the browser context level. The context option is the more useful one: it lets a single browser process hold several contexts with different proxy settings, which is an efficient way to run multiple sessions without paying the cost of a browser launch for each.

Authentication goes in the proxy object as username and password fields rather than in the URL, which avoids awkward escaping. Chromium historically ignores credentials placed in the proxy URL, so use the dedicated fields and you will avoid a frustrating class of silent failure.

Because a proxy is fixed for the lifetime of a context in Playwright, per-request rotation means creating a new context rather than swapping a setting. Fortunately contexts are cheap, and a fresh context also gives you a clean cookie jar and storage state, which is usually what you want when the exit address changes anyway.

Puppeteer

Puppeteer passes the proxy to Chromium as a launch argument, which means the setting applies to the whole browser instance for its lifetime. Rotation therefore means launching a new browser, which is heavier than Playwright's context approach and worth accounting for in your worker design.

Authentication is handled per page through the authenticate method, which responds to the proxy's challenge. Call it before navigating, and be aware it needs calling on each new page. Forgetting this produces a 407 that can look like a credential problem when it is really a sequencing one.

If you need frequent rotation with Puppeteer, the common pattern is a pool of browser instances, each launched against a different proxy port, with work distributed among them. It costs more memory than a single browser with many contexts, but it keeps rotation cheap because you are reusing already-launched instances rather than restarting.

Rotation patterns that hold together

Decide whether the exit address should change per request, per session or on a timer, then implement it in exactly one place. Scattering rotation logic across a codebase is how you end up with cookies from one address being sent from another.

When you do rotate, allow for the reattachment. A mobile modem takes a moment to detach and re-register with the carrier, and requests fired into that window fail in ways that look like network faults. Measure the rotation time on your own ports and build a wait, or a health check, into the rotation step.

Bind session state to the address. When the exit changes, clear cookies, local storage and any cached authentication for that session. A browser context per session makes this automatic, which is a good reason to structure browser work that way even when it seems heavier than necessary.

Debugging failures that all look the same

Work outward from the smallest layer. First confirm the proxy itself works with a plain command line request to an IP echo service, which tells you whether the credentials and port are correct without any framework involved. If that fails, nothing above it will work.

Next confirm your library is genuinely using the proxy. The classic error is a configuration that is silently ignored, leaving requests going out from your own address while everything appears fine. Fetch an IP echo endpoint through the framework itself and compare the answer with the direct one. If they match, your proxy setting is not being applied.

Finally, distinguish a network failure from an application-level block. A 407 is authentication. A connection refused is the wrong port. A timeout after connecting is usually the target, not the proxy. And a 200 containing an interstitial or an empty result is a soft block, which no amount of proxy configuration will fix and which you should handle by slowing down.

• 407: proxy authentication, wrong or missing credentials
• Connection refused: wrong port or protocol for that endpoint
• Timeout after connect: usually the target site, not the link
• 200 with wrong content: soft block, reduce pace before rotating

Frequently asked