Indeed Mobile Proxies for Job Market Data

Job postings are one of the best real-time indicators of what employers need, and Indeed carries a very large share of US listings. Labor economists, recruiting teams and workforce planners use posting counts, titles, pay ranges and locations to read the market. Indeed has also closed its old public job search API and protects its search pages carefully, so collection has to be small, well-structured and polite. This page is a practical guide for data engineers doing that work on a dedicated WeProx US mobile line.

Decide what you are measuring first

Counting jobs, sampling pay and tracking specific employers are three different jobs with very different request budgets. If you need the number of postings for a title in a metro, the search results page shows a count and you may not need detail pages at all. If you need pay ranges, you need the posting where the employer listed one, and many do not. If you need a specific company's hiring pattern, its company page and filtered searches are more efficient than broad sweeps. Pick the narrowest question, because Indeed will not tolerate broad, fast sweeps and nothing about a proxy changes that.

How Indeed search pages behave

Searches combine a query term and a location parameter, with paging controlled by a start offset. A single query stops returning new results after a limited number of pages, so broad searches need slicing by location, date posted, job type or salary filters. Every posting has a job key in its URL; use it as your primary key, because the same job appears in many searches and sponsored placements repeat across pages. Expect duplicates of a different kind too: staffing agencies and multi-location employers post the same role many times with small edits. Decide whether your analysis counts postings or unique roles, and deduplicate accordingly.

Postings expire and are removed. Record first-seen and last-seen dates yourself instead of trusting the displayed age, which is approximate.

Location, country and your IP

Indeed runs separate country sites, and the visitor's IP influences which one you land on and what the default location is. A WeProx line is a US carrier SIM, so you land on the US site with a US default. Results for a given location parameter do not depend on your IP, but personalization and suggested searches can. For regional studies, pass the location explicitly and, if you are auditing what local job seekers see by default, use a line in that metro, for example Chicago, Houston or North Carolina.

Pacing on a mobile line

Indeed challenges traffic that is fast, uniform or missing ordinary browser behavior. A WeProx line removes one negative signal: its address is a US carrier IP shared through CGNAT with many real phones, not a datacenter range. The rest is your client and your pace. Walk each search slice in one sticky session, a few pages at a time with varied pauses, open only the detail pages your question needs, then rotate and move to the next slice. Keep concurrency per line very low on Indeed and scale with lines, not threads.

Timing matters too. Run slices at varied times during US daytime hours rather than in one burst at midnight, and spread a large study across several days. A job that finishes in a week at a modest rate produces the same dataset as one that tries to finish in an hour, and it is far less likely to be interrupted halfway through by a wall of challenge pages that leave holes in the data.

• Do not collect candidate data; resumes and profiles are not public data for harvesting.
• Do not submit applications or interact with employers through automation.
• Stop a session on a challenge page instead of retrying it.

When results look off

A search that suddenly returns far fewer results than yesterday is more often a challenge page or a changed filter serialization than a real market shift; check body size and parameters before publishing a trend. Detail pages missing the description block usually loaded content through a later request your client did not make. Failures clustered just after rotation are the modem re-attaching, not a block.

Setting up a Indeed proxy on WeProx

1. Write down the narrowest question your data must answer.
2. Order a WeProx line, in a specific metro if you audit local defaults.
3. Copy host, port, credentials and rotation URL into your collector.
4. Slice searches by location and date posted to stay within page limits.
5. Key postings on the job key and record first-seen and last-seen dates.
6. Rotate between slices and stop sessions that hit a challenge.

Indeed proxy questions