Python Requests and curl with Mobile Proxies

Python requests and curl are the two tools most data collection work starts with. One is the default HTTP library in Python; the other is on almost every server already. Both accept a proxy URL, both support authentication in that URL, and both can talk SOCKS5 with the right pieces installed. What neither can do is make a server IP look like a phone. That is what a mobile proxy adds. WeProx provides USA 4G and 5G mobile proxies on real carrier SIMs with unlimited data, built for scraping and data collection where volume and consistency matter. This article shows how to form the proxy URL, keep a session on one IP, switch to SOCKS5, add retries, and call the rotation link from the same script.

What requests and curl need from a proxy

Both tools express a proxy as a URL: a scheme, then username and password separated by a colon, an at sign, and the host and port. For an HTTP proxy the scheme is http. For SOCKS5 the scheme is socks5, or socks5h when you want the proxy to resolve hostnames. Python requests reads a dictionary that maps the target scheme, http and https, to the proxy URL. curl takes the proxy URL through its proxy option or the same environment variables requests honours.

Your WeProx dashboard supplies the host, an HTTP port, a SOCKS5 port, a username and a password for each proxy, plus a rotation link. Building the proxy URL is simply assembling those pieces in the right order. If your password contains characters that are special in URLs, encode it, or pass credentials separately as described below.

Python requests through a WeProx proxy

The minimal version is a proxies dictionary with the same URL for http and https keys, passed to a get or post call. Something like proxies = {'http': 'http://user:pass@host:port', 'https': 'http://user:pass@host:port'} followed by requests.get(url, proxies=proxies). Requests handles the CONNECT tunnel for https targets automatically.

For any real collection job, use a Session object and set its proxies attribute once. The session keeps a connection pool to the proxy, reuses TCP connections, and carries cookies between requests, which is what you want when a site expects a consistent visitor. Combine that with a WeProx sticky session and the site sees one carrier IP with one cookie jar for the whole run.

If you prefer not to embed credentials in the URL, requests accepts a proxy authorization header via the auth helper for proxies, or you can read the username and password from environment variables and build the URL at runtime so secrets stay out of source control.

• Same proxy URL for the http and https keys.
• Session object with proxies set once for pooling and cookies.
• Credentials from environment variables, assembled into the URL at runtime.
• Reasonable timeouts on every call so a stalled request does not block a worker.

SOCKS5 with requests

Requests supports SOCKS through an optional extra. Install it with pip install requests[socks], which pulls in the PySocks dependency. Then use a proxy URL with the socks5h scheme and the SOCKS5 port from your dashboard. The h suffix tells the library to send the hostname to the proxy, so DNS resolution happens on the WeProx side rather than on your server.

SOCKS5 is the right choice when you also need UDP, when you want to avoid local DNS entirely, or when a target is sensitive to the way an HTTP proxy rewrites requests. For plain page fetching, the HTTP port is simpler and equally fast.

curl with a mobile proxy

The everyday form is curl with the lowercase x option and the proxy URL, followed by the target: curl -x http://user:pass@host:port https://example.com. For SOCKS5 with proxy-side DNS, use the socks5h scheme in the same option, or the dedicated socks5-hostname option that curl provides. Add the proxy-user option if you want the credentials separate from the URL.

curl is also the quickest way to test a new WeProx proxy: fetch any what-is-my-IP endpoint through it and confirm you see a US carrier address. And it is the natural way to call the rotation link from a shell script or a cron job, since the link is just a URL that returns when the rotation is done.

Environment variables work for both tools. Set http_proxy and https_proxy to the proxy URL and neither requests nor curl needs an explicit proxy argument, which is convenient in containers.

Retries, sessions and rotation between runs

Mobile networks are reliable but not perfectly uniform, so build retries in. With requests, mount an HTTPAdapter that carries a Retry object with a small total count, a backoff factor, and the status codes you want to retry, such as temporary server errors and rate responses. Keep the retry logic in the adapter so every call on the session benefits without extra code.

Decide on a rotation strategy that matches the site. For crawls where each run should come from a fresh address, call the WeProx rotation link at the start of the run, then fetch on a sticky IP for the whole run. Rotations are unlimited with no wait between changes, so you can also rotate after every batch of pages. If you want a different IP on every single request, switch the proxy to per-request mode from the dashboard and leave your code unchanged; the URL stays the same.

The rotation link is a normal HTTP request, so a script can call it with the same session or with curl. Call it, wait briefly for the new IP to be live, and continue.

Scaling and troubleshooting

Use one proxy per worker process or thread pool. Each WeProx proxy is a single modem and SIM, so parallel workers on one proxy share its throughput. For regional coverage, assign proxies by metro: New York, Los Angeles, Chicago, Houston, Phoenix, Miami, North Carolina and Boston, with Boston on 4G LTE only. Location moves are free and self-service from your dashboard when the mix needs to change.

A proxy authentication error means the URL was assembled wrong, usually an unencoded special character in the password or the SOCKS5 port used with the http scheme. Print the URL with the password masked and compare against the dashboard.

A connection refused on the socks5 scheme in requests means the SOCKS extra is missing; install it and try again. Slow responses on a large parallel job usually mean too many workers on one proxy; spread them across more proxies rather than raising the thread count.

If a target reports a non-mobile IP, check that the environment variables are not overriding the explicit proxies dictionary, and that no_proxy does not include the target host.

Why WeProx for scripted collection

Scripts are honest about what they are: they fetch pages. What decides whether a site serves the full page is the address. Ours come from real SIM cards in 4G and 5G modems we own on AT&T, T-Mobile and Verizon, sharing carrier NAT with ordinary phones. That is why a requests session on WeProx receives the same responses a handset does.

Our 4G proxies typically deliver 20 to 45 Mbps and 5G delivers 50+ Mbps. Every proxy includes HTTP(S) and SOCKS5 with UDP, sticky or per-request rotation, a rotation link and an API, plus unlimited data so a long crawl never hits a meter. Plans start from $4/day with daily, weekly and monthly terms.

Frequently asked