A price monitoring proxy pipeline can fail without obvious blocks when proxy pacing drifts and retries cluster into bursts. The tell is not a single 403 or 429, but a sudden drop in field completeness and a spike in cost per usable record. The fix is to treat pacing, session continuity, and retry budgets as queue constraints, not as per-request tweaks.
The situation: coverage looked fine, but the snapshot was unusable
The team was collecting public product pages across several markets. The scraping proxy pool still returned responses, and headline success metrics stayed stable. Yet downstream comparisons started showing false price changes because key fields disappeared intermittently and region-consistent slices became mixed within the same monitoring window.
The pipeline was technically running, but the output could not be trusted for alerts. This is the failure mode that keeps monitoring teams stuck: the queue “works” while the record quality silently collapses.
What changed: pacing drift turned retries into a traffic amplifier
The root cause was a pacing change that increased concurrency during a busy hour. As soon as partial failures increased, retries clustered tightly because the retry policy did not enforce a budget per usable record. That created short bursts that changed response variants and increased structural simplification, which reduced field completeness even when HTTP status stayed 200.
Once the queue entered this bursty retry loop, session continuity became inconsistent. The same target returned different page versions across the monitoring window, producing non-comparable records.
The containment playbook: isolate, cap, replay
- Isolate the monitoring queue: separate monitoring traffic from discovery traffic so pacing is stable.
- Cap retries: define a retry budget per usable record and stop retrying once the budget is hit.
- Replay a short window: run the same query set with the same region consistency rule and session continuity to confirm the failure mode.
When a different proxy strategy is justified
If a controlled replay window still shows unstable field completeness under a strict pacing plan, the problem may be exit volatility rather than pacing. In that case, changing the proxy mix can help, but only after you have a stable queue design that prevents drift from being misdiagnosed.
FAQ
How do I know pacing drift is the real cause?
Run a replayable monitoring window with fixed pacing and session continuity. If field completeness stabilizes and region-consistent slices stop mixing, pacing drift was the primary driver.
Why do retries hurt price monitoring more than discovery?
Monitoring needs comparable snapshots. Bursty retries change timing, session continuity, and page variants within the same window, which produces records that are “successful” but not comparable.
What is a practical retry budget for monitoring queues?
Budget retries per usable record, not per request. Start with a low cap and increase only when replay windows remain stable for several days.