Designing a Scraping Pipeline That Does Not Break

A scraper that works on your laptop and a pipeline that runs unattended for months are different pieces of software. The second one has to assume that targets change, connections drop and pages occasionally lie. Building for those assumptions from the start costs a little time and saves a great deal of it later.

Separate fetching from parsing

The single most valuable structural decision is to keep the fetch stage and the parse stage apart, with raw responses stored in between. Fetching is slow, expensive and spends real time on the wire. Parsing is fast, cheap and the part most likely to need changing when a site adjusts its markup.

Store the raw payload along with the URL, timestamp, response status and the exit location used. When a selector breaks, you re-parse what you already have instead of re-fetching everything, which costs nothing and takes minutes. When someone questions a figure, you can show exactly what the site returned.

This split also makes your pipeline restartable. A crash during parsing loses no network work. A crash during fetching loses only the items not yet stored. Idempotent stages with a durable boundary between them are what let a system recover on its own.

Retries and backoff that stay polite

Not every failure deserves a retry. Classify first: transient network errors and 5xx responses are worth retrying; 404 and 410 are facts and should be recorded as such; 401 and 403 usually mean you are asking for something you should not be asking for, and repeating the request will not change that.

Use exponential backoff with jitter, and cap both the delay and the attempt count. Without a cap, one broken target can occupy a worker indefinitely. Without jitter, a batch of simultaneous failures produces synchronised retries that arrive as a spike at exactly the moment the site is struggling.

Honour explicit instructions. A Retry-After header is the site telling you precisely what it wants, and following it is both correct and effective. Add a circuit breaker per target so that after a run of failures you stop entirely for a cooling period rather than continuing to push.

• Retry transient errors and 5xx; do not retry 404 or 403
• Exponential backoff with jitter, with a hard attempt cap
• Obey Retry-After exactly rather than guessing
• Trip a per-target circuit breaker after repeated failures

Health checks for your connections

Treat each proxy port as a resource with a state, not as a constant. A lightweight periodic check against a stable, cheap endpoint tells you whether the port is reachable, what its current exit address is, and roughly what latency looks like. Record the result and let your scheduler read it.

Take unhealthy ports out of rotation automatically and put them back only after they pass a check, ideally after a rotation to give the modem a chance to reattach to a better cell. This one mechanism prevents the common failure where a single sick connection quietly absorbs a large share of your job queue through repeated timeouts.

Keep health checks cheap and infrequent enough that they do not eat the data budget they are meant to protect. A small request every few minutes per port is plenty, and the historical record it produces is exactly what you need when you are arguing about whether a bad afternoon was your code or the network.

Detecting soft blocks

The dangerous failure is the one that returns 200. A page that renders an interstitial, an empty result list where results should exist, or a truncated body all pass a status check while carrying no useful data. Left undetected, they poison your dataset silently, which is worse than an outright error.

Defend with structural validation on every response. Assert that the elements you depend on exist, that counts fall within a plausible range, and that the response body is above a minimum size. Any assertion failure is a fetch failure and should be logged and retried as one, not passed downstream.

Track the rate of these events over time, per target and per exit location. A rising soft-block rate is the earliest warning you get that your pacing has become too aggressive or that a target has changed its defences. Reacting to the trend is much cheaper than reacting to a day of empty data.

Monitor the numbers that predict trouble

Raw error counts tell you little. Rates and distributions tell you a lot. Track success rate per target, latency percentiles rather than averages, bytes transferred per run, records extracted per run compared with the previous run, and soft-block rate per exit location.

Alert on change rather than on absolute thresholds where you can. A target whose extraction count falls by a third overnight is far more informative than one that crosses an arbitrary line, and it catches the failures that never produce an error at all.

Watch your byte counters explicitly even with unlimited data. A change on the target side such as heavier pages or a new video element can multiply your transfer volume and stretch run times far beyond plan, and the byte counters are the first place that shift becomes visible.

Validate the data, not just the pipeline

A pipeline can be perfectly healthy and still produce nonsense. Add checks on the output itself: types and ranges on every field, required fields present, sensible bounds on values, and duplicate detection across runs.

Compare against history. If a price field is suddenly null on most records, or a category that always returns hundreds of items returns three, something changed and it is probably your parser rather than the world. Set these comparisons to fail the run loudly, because a quiet partial dataset is the most expensive outcome of all.

Keep the provenance attached through to the final store: source URL, fetch timestamp, exit city and parser version. When an analyst queries a number six weeks later, that metadata turns a long investigation into a short one, and it is the cheapest thing in the whole pipeline to add.

Frequently asked