Proxy pacing diagnostics should start when field completeness drops while public pages still return successful responses. The workflow is for crawler reliability teams running authorized public data collection, price monitoring, or SERP monitoring; it does not fit private targets or tasks that lack stored response, market, and retry records.
Successful responses can still be unusable
A public page may return a response while price, availability, snippet, or source fields are missing. Treat this as a record quality issue before changing proxy volume.
The first step is to compare complete and incomplete records by market, proxy lane, collection time, retry count, and parser status. A pacing change often shows up as a field pattern before it becomes a visible failure.
Isolate the queue that changed most recently
Field drops usually follow a queue change, a retry rule change, or a market lane mix. Freeze unrelated queues and replay a small sample from the affected lane.
| Signal | Likely cause | Next action |
|---|---|---|
| Incomplete fields after retries | Retry pressure is crowding the lane | Lower retry count and lengthen backoff |
| One market drops more than others | Market lane drift or mixed routing | Separate the market lane and replay a sample |
| Parser status changed with pacing | The response shape changed under load | Compare stored source snapshots before parser edits |
Replay before rewriting the parser
Parser changes are expensive when the real issue is proxy pacing. Replay the same public URL set through a slower lane with the same market, then compare required fields.
If fields recover, keep the parser stable and adjust pacing. If fields remain missing, review source snapshots and update parsing rules with evidence from visible page changes.
Restore traffic in small windows
After a fix, restore traffic by queue and market rather than all at once. Watch field completeness, retry share, response time, and cost per usable record for at least one full collection window.
The goal is to recover trustworthy public records, not to return to the previous request rate as quickly as possible.
FAQ
Why can proxy pacing reduce field completeness without obvious failures?
Fast retries or mixed lanes can return successful responses with incomplete public fields, so status codes may look healthy while usable records decline.
Should parser changes come before proxy pacing checks?
No. Replay the affected public URLs through a slower, consistent market lane first; change the parser only when stored source snapshots show a visible page change.