This scenario appears when a team mixes baseline monitoring traffic with discovery traffic in one queue to “keep it simple”. The queue still returns pages, but field completeness drops and region signals drift, so the snapshots stop being comparable. The fix is isolation and limits, not more raw proxy volume.
How this scenario usually appears
A baseline queue starts stable: fixed URLs, fixed region rules, predictable pacing. Then coverage pressure arrives. The team adds more URLs and more workers into the same queue, and failures begin to retry immediately.
Within days, the queue becomes noisy. Reports show inconsistent outputs. The team cannot tell whether the market changed or the pipeline drifted.
Factors that make the issue worse
Unlimited retries amplify burstiness. Mixing markets in the same queue amplifies drift. Sharing capacity between baseline and discovery traffic amplifies both because discovery steals budget from the baseline at unpredictable moments.
Why this setup is more stable
Split baseline monitoring from discovery. Give each market its own queue with one region rule. Enforce a pacing envelope. Cap retries by count and by time. This keeps failures from reshaping the queue and keeps the baseline comparable.
Signals that show whether it worked
Watch region sentinel consistency and field completeness first. Then track cost per usable record and retry budget usage. If the baseline stays stable across runs, you can expand discovery coverage safely without contaminating the baseline.
FAQ
Why does mixing traffic break comparability?
Because it changes pacing and capacity across runs. Even small changes can shift region signals and reduce field completeness.
Should discovery queues share exits with the baseline?
They can, but keep boundaries stable and do not share the same pacing budget. Isolation matters more than variety.
What is the quickest containment action?
Separate baseline and discovery queues and cap retries. That stops the self-amplification loop.
