When login and cart flows break during scraping, the fix is rarely “rotate faster”. The stable setup is to pin a session to one exit for a short, measurable window, rotate only on explicit failure signals, and prove continuity with a few deterministic checks (cookie persistence, redirect loops, and field completeness) before you scale concurrency.
Start from the target workflow, not the proxy pool
Map the exact steps the site expects: landing → login → post-login redirect → cart/search/action. A “working” request that only loads the homepage can hide the real failure point.
For each step, list what must remain consistent: cookie jar, device fingerprint surface (headers and TLS profile), and the exit IP / ASN / region if the site is sensitive to location drift.
Proxy and session rules that keep continuity intact
Use a rotating residential proxy, but do not rotate on every request. Instead, treat a session as a short lease (for example 3–10 minutes) and keep the same exit for all requests within that lease.
Rotate only when you see a continuity break: repeated 302 back to login, a sudden CAPTCHA wall, or a sharp drop in required fields (price, availability, shipping options) compared with the baseline.
If you must distribute load, spread sessions across exits, not requests across exits. This keeps per-session behavior coherent and reduces the chance of cross-exit correlation flags.
Signals to detect continuity breaks early
Track a small set of cheap checks: login success marker, cart item count, and one stable page element that should not disappear (for example, the account menu).
Also track data quality signals: missing JSON keys, empty arrays where you normally get items, or localized content suddenly switching language. These often show a soft block before hard 403/429 appears.
Rollout checklist for production scraping
Start with one session, verify correctness end-to-end, then increase sessions gradually. Keep a backoff policy so one failing exit does not trigger a cascade of retries.
Log per-session outcomes (success, soft-block, hard-block) and rotate only the sessions that are failing. This reduces churn and cost while improving repeatability.
FAQ
Should I rotate on every request to look “more human”?
No. For authenticated or stateful workflows, request-level rotation often breaks continuity and increases anomalies.
What is the simplest continuity test after login?
Request a post-login page that requires authentication and verify a stable marker plus expected redirect behavior.
How long should a session stay pinned to one exit?
Long enough to finish the workflow reliably, short enough to rotate away from deteriorating exits based on your failure signals.